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Part One 



Overview of the Local Systemic Change Initiative 



In 1995, the National Science Foundation (NSF) initiated the Local Systemic Change through 
Teacher Enhancement program. The initiative’s primary goal is to improve instruction in 
science, mathematics, and technology through teacher professional development within whole 
schools or school districts. NSF funded the first cohort of Local Systemic Change (LSC) 
projects in 1995, and an additional cohort of projects each year, for a total of 88 projects funded 
by 2002. 

The LSC initiative distinguishes itself from former NSL-supported teacher enhancement efforts 
in two important ways. Lirst, it targets all teachers in a jurisdiction for professional 
development; each targeted teacher is to participate in a minimum of 130 hours of professional 
development over the course of the project. 1 Second, the LSC emphasizes preparing teachers to 
implement district-designated mathematics and science instructional materials in their classes. 

In addition to providing professional development for teachers, the LSC initiative promotes 
efforts to build a supportive environment for improving science, mathematics, and technology 
instruction. LSC projects are expected to align policy and practice within targeted districts, and 
to engage in a range of activities to support reform, including: 

• Building a comprehensive, shared vision of science, mathematics, and technology 
education; 

• Conducting a detailed self-study to assess the system’s needs and strengths; 

• Promoting active partnerships and commitments among an array of stakeholders; 

• Designing a strategic plan that includes mechanisms for engaging teachers in high 
quality professional development activities over the course of the project; and 

• Developing clearly defined, measurable outcomes for teaching, and an evaluation plan 
that provides formative and summative feedback. 



1 Prior to 1999, NSF required only 100 hours for teachers in K-8 projects. 
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The Core Evaluation 



NSF’s solicitation for the LSC initiative indicated the Foundation’s interest from the outset in 
providing a framework for collecting data from LSC projects to evaluate their efforts. The goal 
of the evaluation activities was not only to assess individual projects, but also to aggregate data 
across projects to glean broader insights about the design, quality, and impact of the LSCs. NSF 
contracted with Horizon Research, Inc. (HRI) in Chapel Hill, North Carolina to develop a data 
collection framework, to provide technical assistance in implementing evaluation activities, and 
to prepare cross-site analyses of evaluation results. 

Since the LSC’s inception, HRI has collaborated with NSF staff, LSC Principal Investigators 
(Pis), and project evaluators on the design and implementation of a core evaluation system. The 
system includes the collection of baseline data during an LSC’s first year of funding, and a range 
of data collection activities during subsequent years. Evaluators are asked to provide 
comprehensive evaluation reports in the second and final years of their projects, and less detailed 
reports in the interim years. 

All of the evaluation activities are driven by a set of core evaluation questions: 

> What is the overall quality of the LSC professional development activities? 

> What is the extent of school and teacher involvement in LSC activities? 

> What is the impact of the LSC professional development on teacher preparedness, 
attitudes, and beliefs about mathematics and science teaching and learning? 

> What is the impact of the LSC on classroom practices in mathematics and science? 

> To what extent are the district and school contexts becoming more supportive of the 
LSC vision for exemplary mathematics and science education? 

> What is the extent of institutionalization of high quality professional development 
systems in the LSC districts? 

Data Collection 

While LSC projects are bound by goals and expectations set by NSF, they also must develop a 
strategy that is responsive to local context and needs. Although the result has been a set of LSC 
initiatives that range considerably in design and implementation strategies, HRI has created an 
evaluation system that allows information to be aggregated across these diverse LSC projects. 
Project evaluators are asked to collect data using standardized questionnaires and protocols 
designed to answer core evaluation questions, and to complete ratings on the quality of LSC 
professional development programs. Data collection activities completed by evaluators of each 
LSC project include the following: 

• Observations of professional development activities; 
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• Classroom observations; 2 

• Teacher questionnaires; 

• Principal questionnaires; and 

• Teacher interviews. 

Additional data collected from the LSCs included: 

• HRI Interviews with LSC Principal Investigators; 

• Project ratings completed jointly by evaluators and Pis; and 

• Project strategies questionnaires completed jointly by evaluators and Pis. 

• Observations of Professional Development Activities 

The core evaluation requires evaluators to conduct observations of professional development 
sessions each year, and to record their observations on standardized protocols. Established 
projects are asked to conduct 5-8 observations, while baseline projects conduct at least two 
observations. Evaluators are to consult with LSC Pis on the professional development activities 
planned throughout the year, and select a sample that is representative of the range of 
experiences offered. Program-wide, over 2,400 observations of professional development 
sessions were conducted during the years reflected in this report. Data were weighted to control 
for the variable number of observations conducted per project. 

• Classroom Observations 

HRI provided the lead evaluator in each project with a list of randomly selected teachers for each 
targeted subject. These teachers, or their randomly selected back-ups, were to be observed in the 
spring, at the beginning, midway, and end of the project. Roughly, 70 percent of teachers 
selected for the original sample agreed to be observed; these teachers tended to have somewhat 
higher levels of participation in LSC professional development than those teachers that declined 
to be observed (Bowes & Banilower, 2004). A total of 1,620 lessons were analyzed for this 
report; about 55 percent of these lessons were taught by teachers who had participated in at least 
20 hours of LSC professional development. In all cases, the data were weighted to represent the 
total population of eligible teachers in the project. 

• Teacher Questionnaires 

Each project was asked to administer questionnaires to a random sample of teachers for each 
targeted subject. Teacher questionnaires asked respondents about attitudes, beliefs, preparedness 
to teach, instructional practices, and so on. Nearly 75,000 teacher questionnaires were returned 
to HRI, including 36,828 from K-8 science teachers; 24,903 from K-8 mathematics teachers; 
11,206 from 6-12 mathematics teachers; and 2,021 from 6-12 science teachers. Weights were 
added to the data file to reflect the probability of each teacher’s selection into the sample, 
adjusted for any non-response in that project. Projects were expected to achieve an 80 percent 
response rate for the teacher questionnaires; between 2000 and 2004, the median response rate 
was 83 percent. 



2 In projects targeting both mathematics and science, or both elementary and secondary mathematics, questionnaire 
and observation data were collected separately for each “subject.” 
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• Principal Questionnaires 

Projects were also asked to administer questionnaires to the entire population of principals of 
targeted schools each year. Respondents were asked their opinions about mathematics and 
science instruction, factors affecting mathematics/science instruction in their school, and their 
school’s progress in implementing LSC reforms. A total of 17,380 principal questionnaires were 
used in the analyses for this report. For principal questionnaires, projects were expected to 
achieve a response rate of 90 percent; between 2000 and 2004, the median response rate was 
greater than 95 percent. 

• Teacher Interviews 

Evaluators in each of the projects were asked to interview a sample of teachers who had 
participated in at least 20 hours of LSC professional development activities. Roughly 75 percent 
of the teachers selected for the original sample agreed to be interviewed; the other teachers 
interviewed were from the backup sample. A total of 1,782 interviews in 76 projects were used 
for analyses in this report. Evaluators summarized the interview data by completing a summary 
form with both ratings and quotations from the participating teachers. Interview data from each 
project were weighted to reflect the total number of teachers who had participated in the LSC 
professional development in that project. 

• Interviews with LSC Principal Investigators 

In addition to the core evaluation activities described above, HRI conducted in-depth interviews 
with Pis of ending LSC projects in 2000, 2001, 2002, and 2003. Interviews conducted in the 
spring of 2000 with 12 Pis focused on “lessons learned” around professional development 
providers, professional development activities, and sustaining reform. Interviews with 16 Pis in 
the spring of 2001 7 and 19 additional Pis in the spring of 2002, focused on stakeholder and 
policy support for LSC reforms, and issues related to institutionalization. Interviews with 18 
LSC Pis in 2003 focused on issues relating to strategic leadership. 

• Project Ratings 

Beginning in 1998, project Pis and evaluators were asked to jointly complete a set of project 
ratings. Each year, the evaluators and Pis submitted ratings of the quality of their LSC 
professional development programs. Ratings of the supportiveness of the context for the LSC 
vision and likelihood of institutionalization of the LSC reforms were submitted in projects’ 
Baseline Year, Year Two, and Linal Year of data collection. Data from 85 projects at multiple 
time points were used for analyses in this report. 

• Project Strategies 

In 2001, 2002, and 2003, evaluators and Pis of LSCs that were still active were asked to 
complete a project strategies questionnaire. The purpose of the questionnaire was to collect 
information on the design of the professional development program of each LSC, including the 
types of professional development providers utilized, the foci of professional development, and 



3 Eleven of these Pis had been interviewed in 2000 as well; 5 Pis represented projects that had “officially” ended 
prior to 2000. 
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how professional development time was allocated. Seventy projects completed this 
questionnaire at least once over this time span. 

Training Evaluators 

Lead Evaluators, the member of the LSC evaluation team responsible for coordinating the core 
evaluation data collection and writing the annual evaluation report, and other members of the 
LSC evaluation teams were required to attend meetings on data collection for the core 
evaluation. These meetings provided orientation to the instruments being used for program- wide 
data collection, as well as training in their appropriate use. Subsequent annual meetings with 
Lead Evaluators provided “refresher” sessions in the use of many of these instruments. 

As part of these meetings, Lead Evaluators were trained in the use of core evaluation instruments 
to assess the quality of professional development sessions/programs. The training provided 
evaluators opportunities to gain experience with the core evaluation instruments using videotapes 
and vignettes focused on key issues related to the quality of professional development. Lead 
Evaluators were also provided with materials they could use to conduct training sessions with 
other members of their evaluation team. 

Evaluators also received training on the use of the LSC Classroom Observation Protocol. Prior 
to conducting classroom observations, prospective observers were required to participate in a 
two-day training session that consisted of observing and rating videotaped classroom lessons. 

The purpose of the training was for LSC observers to gain understanding of the standard “rating 
keys” developed by a norming group of science and mathematics educators and to learn to rate 
lessons in accordance with the rating keys. It was necessary for all classroom observers to 
demonstrate appropriate calibration and reliability on the LSC Classroom Observation Protocol 
in order to conduct classroom observations for the core evaluation. 

Data Analysis and Reporting 

Analyses of the impact of the LSC initiative on teachers and their teaching are typically reported 
by extent of teacher involvement in LSC professional development activities. Differences noted 
in this report are statistically significant at the 0.05 level. Lull descriptions of the research and 
analysis methodologies are available in the technical reports cited throughout this report. 



Purpose of the Capstone Report 

NSL’s Local Systemic Change program is in its final stages, with 18 of the 88 funded projects 
still active. Over the last decade, the initiative has left its mark on teachers, classrooms, schools, 
and districts. The purpose of the Capstone report is to look at the impact of the LSC from both a 
cross-site and longitudinal perspective. In particular, key findings from the analysis of core 
evaluation data provide some insights into topics relevant for other large-scale reform 
efforts — for example, the selection and preparation of professional development providers; 
designing effective interventions; reaching targeted audiences; and strategies for building 
stakeholder, policy, and “system” support for reforms. We believe that these findings will be of 
use to those who are designing and/or leading similar kinds of reforms that seek to influence 
classrooms and systems alike. 
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Part Two 



Overview of LSC Projects 



The LSC projects served a wide variety of districts, schools, and students. The initiative funded 
a total of 88 projects: 38 K-8 science projects, 6 secondary science projects, 18 K-8 
mathematics projects, 14 secondary mathematics projects, 6 projects that targeted both 
elementary mathematics and science, 1 project that targeted both elementary and secondary 
science, and 5 projects that targeted both elementary and secondary mathematics. Sixty-six of 
the projects were funded as five-year projects, 14 as four-year, and 8 as three-year; although a 
number of projects were granted no-cost extensions by NSF that extended their duration. 

Thirty-eight of the LSC projects were single-district projects; at the other end of the scale, 4 
projects involved more than 20 districts each. The 88 projects involved approximately 70,000 
teachers in roughly 4,000 schools in 467 districts across the United States. By the completion of 
these projects, an estimated 2,142,000 students will receive instruction from LSC-treated 
teachers each year. 

As can be seen in Figure 1, 49 percent of the schools targeted by the LSC are in urban areas, 25 
percent are in suburban areas, 13 percent are in rural areas, and 14 percent are in towns or small 
cities. 



Percentage of LSC Schools in 
Various Types of Communities 



Rural 

13% 




25 % 



Figure 1 



In terms of student demographics, across all schools targeted by the LSCs, 48 percent of students 
are white, 23 percent African-American, 21 percent Hispanic, 6 percent Asian, 1 percent 
American Indian or Alaskan Native, 0.4 percent Native Hawaiian or Pacific Islander, and 0.4 
percent are from another background. As can be seen in Figure 2, projects targeting K-8 
mathematics serve the largest proportion of minority students, but in each subject the 
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representation of minority students is about as large as, if not larger than, the national average of 
approximately 40 percent. 

The typical school targeted for K-8 mathematics or science reform by the LSC projects has 530 
students, 50 percent of whom qualify for free or reduced-price lunches and 12 percent of whom 
are of limited English proficiency (LEP). The typical school targeted for 6-12 mathematics or 
science reform has 762 students, 43 percent of whom are eligible for free or reduced-price 
lunches and 8 percent of whom are classified as LEP. 
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Figure 2 
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Theory of Action 

The LSC theory of action argues that providing teachers with opportunities to deepen their 
content and pedagogical knowledge in the context of high quality instructional materials will 
result in better-prepared teachers. The theory also predicts that these teachers, with ongoing 
support, will be more inclined to change their instruction in ways advocated by national 
standards, and will have more capacity to do so. Improved instruction will in turn lead to higher 
student achievement . 4 
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Figure 3 



Many factors influenced the design of LSC projects: the background and experience of key 
project staff; the history of prior reform activities in the district(s); district size and urbanicity; 
the involvement of a single district versus multiple districts; the types of partners involved in 
reform efforts; and so on. Whatever the individual design, however, LSCs were guided by a 



4 Examining the link between improved instruction and student outcomes was not required of projects, or the core 
evaluation, until the fifth cohort of projects. Thus, the arrow representing this link in Figure 1 is dashed rather than 
solid. 
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common set of principles to achieve their overarching goal of improving mathematics/science 
instruction. These principles included the following: 

• Using well-prepared professional development providers whose backgrounds include 
in-depth content understanding and expertise in K-12 mathematics/science education; 

• Establishing a supportive and collegial professional development culture that 
facilitates teacher learning; 

• Providing experiences that deepen teachers’ knowledge of the mathematics/science 
content in the curriculum and the pedagogy needed to teach this content; 

• Providing opportunities for teachers to explore and become conversant with high- 
quality instructional materials and the appropriate pedagogy for using these materials 
in their classrooms; and 

• Providing support for teachers in content, pedagogy, and materials over the course of 
implementation. 

The next two sections describe some of the approaches that LSC projects have used to 
operationalize the LSC theory of action. First, we take a look at the design and format of LSC 
professional development activities; the use of professional development providers; attention to 
content, pedagogy, and materials; and support for teachers. We then describe some of the LSC 
strategies used to gamer stakeholder and policy support for institutionalizing the reforms 
initiated with NSF funds. 



Approaches to Professional Development 

LSC designs for professional development typically include summer institutes over a period of 
one or more weeks, and professional development and follow-up support offered during the 
school year. School-year offerings may include a range of activities: workshops or seminars, 
content institutes, study groups, classroom demonstrations/observations/debriefings, coaching, 
and mentoring. Whether centralized or school-based, these professional development activities 
may have different purposes and may vary in their format and focus, depending on the needs of 
targeted teachers and the types of professional development providers used. 

Design and Format of LSC Professional Development 

LSCs are expected to provide targeted teachers with 130 hours of professional development 
around mathematics/science. On the project strategies questionnaire, 87 percent of responding 
projects indicated that a typical teacher would take three or more years to reach the LSC target of 
130 hours; roughly one-quarter of the LSCs expected teachers to need five or more years to reach 
the target. (See Table 1.) A small percentage of LSC projects expected teachers to receive 130 
hours of professional development in two or fewer years. 
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Table 1 

Number of Years It Takes for a Typical Teacher to 
Accumulate 130 Hours of LSC Professional Development 





Percent of Projects 
(N = 70) 


1 Year 


3 


2 Years 


10 


3 Years 


30 


4 Years 


30 


5 or More Years 


27 



Nearly half of LSC professional development time was devoted to engaging teachers in 
mathematics/science inquiry. (See Table 2.) Typically, teachers “experience” the activities as 
students might, i.e., working through problems in small groups, with guided discussions 
facilitated by project staff, teacher leaders or content experts. Receiving information, examining 
classroom practice, and examining student work each represented a smaller portion of teachers’ 
professional development time. 



Table 2 



Projects’ Design for Professional Development: 
Format of Professional Development 





Percent of Professional Development Hours 
(N = 70) 


Minimum 


Maximum 


Mean 


Standard 

Deviation 


Engaging teachers in mathematics/science inquiry 


5 


80 


45.86 


18.77 


Presenting information to teachers 


0 


55 


17.07 


12.55 


Examining classroom practice (e.g., observation/coaching, videos) 


0 


60 


16.14 


12.28 


Examining student work 


0 


40 


11.14 


7.53 


All other formats 


0 


70 


9.79 


13.09 



As shown in Table 3, for typical teachers, about one-half their LSC professional development 
hours occurred in a large group setting; about 40 percent took place in groups of fewer than 15 
participants — a strategy likely designed to promote a professional development culture 
conducive to teacher learning, reflection, and dialogue, and to allow more in-depth exploration of 
content, pedagogy, and materials. Often, these small group sessions occurred in grade-level 
meetings within or across schools. LSCs devoted considerably less time to individual or one-on- 
one work with teachers; given the amount of resources required for these kinds of activities, it is 
not surprising that only 10 percent of teachers’ professional development time was spent in this 
way. 
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Table 3 



Projects’ Design for Professional Development: 
Size of Professional Development Activities 





Percent of Professional Development Hours 
(N = 70) 


Minimum 


Maximum 


Mean 


Standard 

Deviation 


In large group meetings/activities (15 or more participants) 


5 


85 


49.93 


21.51 


In small group meetings/activities (fewer than 15 participants) 


5 


90 


39.64 


20.63 


In one-on-one observation/coaching 


0 


80 


10.43 


11.85 



Targeted teachers included those with diverse backgrounds and needs. Some LSCs offered a 
range of formats, topics, and levels — an approach designed to “capture a broad stratification of 
teachers” over time, and meet needs in a carefully sequenced design. Given that the majority of 
LSCs could not mandate teacher participation, ensuring that teachers participated in the kinds of 
professional development activities that best prepared them for implementation presented a 
major design challenge. On the average, teachers were able to select from a variety of activities 
for about one-third of their professional development hours; however, most of their professional 
development time was spent in activities designated by the LSC, typically beginning with 
introductory workshops that focused on the designated instructional materials and inquiry-based 
methods of instruction. (See Table 4.) 



Table 4 



Projects’ Design for Professional Development: 
Amount of Choice in Professional Development Activities 





Percent of Professional Development Hours 
(N = 70) 


Minimum 


Maximum 


Mean 


Standard 

Deviation 


Activities designated for the teacher by the LSC 
Activities offered by the LSC, but as an option or elective 


0 

0 


100 

100 


69.50 

30.50 


29.64 

29.64 



Evaluators suggest that LSC designs using a “multi-tiered” approach are particularly effective for 
meeting teachers’ needs, providing a coherent and “developmental” approach, in which project 
leaders proactively guide teachers in the choices they make that results in a rich and varied 
“sequence of professional experiences that build on one another.” The strategy typically began 
with materials-based workshops, ranging from introductory to intermediate-level explorations of 
how to effectively implement the designated instructional materials. As teachers moved beyond 
introductory sessions, evaluators noted that more advanced, “second level” professional 
development activities gave greater emphasis to inquiry, constructivism, student thinking and 
learning, the needs of diverse learners, and assessment strategies — all topics that may have been 
introduced in earlier sessions. Said these evaluators of this kind of approach: 

All professional development is about the curriculum, although project staff have made a 

distinction between two levels of workshops. While both focus on the curriculum, Level 
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One workshops are more explicitly about orientation to the materials and the 
mathematics content. Level Two workshops are more about the pedagogy used in the 
curriculum. (These differences are a matter of degree, since pedagogy is a part of Level 
One workshops, too.) In the last year of the project, more emphasis has been placed on 
offering Level Two workshops. (Evaluator, K-12 mathematics LSC) 

The program has been developmental; it has embraced the research on ways to best 
address teachers ’ needs based on their level of use of a new educational innovation. The 
project has continued to deepen teachers ’ knowledge of, and experience with, 
appropriate pedagogy to develop students’ conceptual understanding of science past the 
curriculum implementation stage and through its ongoing PD program. (Evaluator, K-8 
science LSC) 

The lack of this kind of developmental approach could reduce the overall effectiveness of LSC 
professional development programs. For example, evaluators described LSC projects that were 
less coherent in their design, with teachers receiving little guidance in their selection of 
workshops that might best meet their needs. Further, some of the designs lacked a deliberate 
strategy for monitoring needs and a proactive approach for meeting these needs, relying instead 
on a laissez-faire approach dependent on the “interest and initiative of the individual.” 

As shown in Table 5, about half of the LSC projects planned to meet the diverse needs of 
participants within each professional development activity. About one-third of projects decided 
to have teachers participate in a common core of activities, plus a small set of activities tailored 
to various levels of expertise. Roughly 1 in 7 projects had most of their activities tailored for 
teachers of different levels of expertise. 



Table 5 



Projects’ Design for Professional Development: 
Accommodation of Teachers with Different Levels of Expertise 





Percent of Projects 
(N = 70) 


Each professional development activity is designed to accommodate teachers with a range of 
different levels of expertise 


49 


Teachers participate in many of the same core professional development activities, and have few 
activities tailored to their different levels of expertise 


37 


Teachers participate in few of the same core professional development activities, and have many 
activities tailored to their different levels of expertise 


14 



Evaluators provided examples of the ways in which LSCs tweaked the format or design of the 
professional development to better meet teachers’ needs as projects unfolded. For example, in 
response to teacher surveys in new LSC schools, staff in one project designed “extension” 
workshops, seminars, and school-based “Learning Forums” to address “trouble spots.” Other 
LSCs redesigned summer workshops to better address the needs of both beginning and 
experienced teachers. For example, when attendance was poor at curriculum training workshops 
during the school year, one project developed summer mini-courses that addressed relevant 
topics, including assessment, ways to extend investigations, and the use of student journals. 
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Professional Development Providers: Roles, Preparation, and Support 
LSCs typically tap a diverse group of professional development providers with a range of 
backgrounds and experience in content, teaching, and reform leadership. Included among these 
providers are: 

• Teachers on special assignment, released full time from their classroom 
responsibilities; 

• Regular classroom teachers, who are providing professional development in addition 
to their full- or part-time classroom responsibilities; 

• District-level personnel, including mathematics/science supervisors; 

• College/university mathematics/science faculty and/or business industry scientists; 

• College/university mathematics/science education faculty; and 

• Other non-district personnel (e.g., museum personnel or textbook representatives). 

LSCs often sought out professional development providers whose vision of mathematics/science 
instruction already matched that promoted by the LSC — providers who brought a “strong 
understanding” of the curriculum, who were experienced professional development facilitators, 
and who complemented the skills and background of project team members. LSCs tapped 
university and/or industry scientists and mathematicians specifically for addressing the content 
needs of participants. This evaluator’s comment describes the ways in which content experts 
were typically used in LSC projects: 

Deepening content knowledge was the primary charge for an experienced and dedicated 
group of Project Scientists, most of whom came from the science faculty at [University]. 
They included specialists in science, environmental science, physics and astronomy, 
geology, chemistry, and biochemistry. . . These professionals have exceptional credentials 
and experience in their science specializations and they comprise the core of what has 
been a very effective cadre utilized to help teacher participants increase their knowledge 
and deepen their understanding of science content. (Evaluator, K-8 science LSC) 

Teacher leaders also played significant roles in many LSCs. Teachers on special assignment 
(TSAs) released full-time from their teaching assignments planned activities for targeted 
teachers; facilitated district, project-wide, or school-based professional development; and 
provided coaching and mentoring to individual teachers. As former teachers, TSAs bring 
classroom experience and credibility to reform efforts; many also bring prior experiences as 
professional development providers, and were selected by LSC staff precisely because they have 
already distinguished themselves as leaders in their schools and districts. Some LSCs also used a 
second tier of teacher leaders — those who maintain their classroom teaching assignments during 
the LSC. These leaders typically assisted more experienced professional development providers 
in facilitating workshops, and provided on-site support to their colleagues. 

LSCs often used an approach that paired university faculty and teacher leaders. Within this 
teaming format, teacher leaders typically focused on pedagogy and materials, while content 
experts worked to develop conceptual understanding. The following example is typical of the 
ways in which providers shared responsibility for different areas of professional development: 
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Teams of two or more instructors shared responsibility for the design and delivery of the 
workshops. Typically, the entire team addressed issues of pedagogy and materials 
management while one member, selected for strong content knowledge, took the lead in 
that area, [offering] mini-lessons around key concepts being addressed in the materials. 
But even these responsibilities were shared in many of the workshops. So the roles were 
less clearly defined than the designation of “content expert ” might imply. This blurring 
of functioned responsibilities increased over time as team members gedned expertise. 
(Evaluator, K-8 mathematics/science LSC) 

The ability of the “provider” team to address content, pedagogy, and leadership was cited by one 
LSC leader as the “biggest asset in arriving at a balanced professional development program.” 

At the same time, another LSC PI noted that using this strategy posed challenges in developing 
mutual trust and respect between faculty and teacher leaders, often taking a number of years to 
establish these partnerships and a level of camaraderie that was effective with targeted teachers. 

LSC professional development providers assumed a wide range of roles — workshop facilitation, 
materials review, mentoring and coaching, materials management, community outreach, and 
advocacy. While content experts needed little preparation in subject matter, some had limited 
experiences around pedagogy and in working with K-12 teachers. Some LSCs addressed these 
shortcomings more extensively than others by engaging mathematicians and scientists in 
intensive training. In one LSC, activities designed to help scientists work more effectively with 
targeted teachers included “critical friend feedback” by mentors, and debriefings facilitated by a 
workshop “recorder” who attended each professional development session. In contrast, other 
LSCs relied only on brief discussions with content experts to prepare them. 

Teachers on special assignment typically participated in the most extensive preparation in terms 
of building knowledge and skills. Among the strategies used were: intensive summer institutes 
to familiarize teacher leaders with content, pedagogy, and materials; leadership institutes; 
modeling/mentoring/coaching/debriefing by core staff; study groups and action research teams; 
and consultant-led sessions on specific topics such as inquiry, equity, and assessment. TSAs also 
participated with LSC Pis and project directors in national conferences and/or institutes to gain a 
better understanding of standards, effective professional development, and systemic reform 
strategies. 

Preparation for “second tier” part-time teacher leaders included many of the same strategies used 
with TSAs — typically focusing first on the designated instructional materials and the content and 
pedagogy that under-gird those materials, but over time expanding these experiences to provide 
more in-depth learning opportunities both locally and nationally. One LSC used a model that 
included a series of stages to groom teacher leaders and help them grasp new practices — where 
leaders first attended “basic” workshops, then observed and scripted workshop sessions, then 
participated in planning and leadership sessions, and finally, joined the core team to design and 
present sessions. 

Involvement in regularly scheduled staff meetings throughout the year was the primary 
mechanism for defining roles, building knowledge and skills, and providing support for 
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professional development providers. In an urban LSC, for example, monthly staff meetings 
helped improve communication among project mentors, and provided opportunities to discuss 
problems and devise solutions to implementation barriers. One LSC “reframed” and designed a 
“coherent, year-long set of experiences” for inexperienced teacher leaders who expressed 
confusion about roles and expectations early in the project. Other LSCs prepared written 
guidelines or incorporated a more participatory process. 

LSCs also supported professional development providers through mentoring, debriefing, 
leadership training opportunities, and the teaming of veteran and less experienced providers. In 
one LSC, opportunities for co-planning, co-facilitation, and debriefing were, in the words of one 
evaluator, “for the explicit purpose of mentoring” facilitators to better support them as they 
conducted professional development sessions on their own. Some LSCs created “communities 
of learners” to develop a system of mutual support among professional development providers. 
Other projects amended budgets, added administrative staff to assist teacher leaders, and worked 
with districts to increase their level of support. 

LSCs that had worked for a number of years with professional development providers — through 
the LSC and previously funded projects — often exhibited more highly developed strategies for 
supporting these providers. Some closely monitored providers’ needs, refined tasks and roles, 
and provided continuing opportunities for providers to hone vision and skills. For example, one 
“experienced” LSC developed an extensive, integrated system for both preparing and supporting 
professional development providers, including formal and informal meetings, institutes and 
conferences, co-teaching with peers, mentoring by core staff and master teachers, study group 
research, and development of teaching “scripts” for workshops. In another project, “Alumni 
Institutes” for leaders who had already attended core training provided additional opportunities 
to deepen knowledge and skills in content, pedagogy, leadership, and facilitation. 

Attention to Content, Pedagogy, and Materials 

Evaluator reports suggest that LSCs have tried to weave content, pedagogy, and the treatment of 
instructional materials together in a “seamless” fashion, using the modules/kits as the “hook” for 
conveying content and pedagogical knowledge. About half of the LSCs emphasized the 
implementation of the designated instructional materials and pedagogy as opposed to 
mathematics/science content in professional development; 11 percent of the LSCs placed a 
greater emphasis on content. (See Table 6.) Table 7 paints a similar picture, a typical teacher 
would spend about one-third of professional development time to deepening knowledge of 
effective pedagogy and one-third to increasing understanding of the instructional materials. 
However, projects also spent nearly a third of professional development time on deepening 
teachers’ content knowledge, suggesting that LSCs tried to give balance to these three 
dimensions. 
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Table 6 

Projects’ Design for Professional Development: 
Emphasis of Content and Pedagogy/Instructional Materials 





Percent of Projects 
(N = 69) 


Emphasized content far more than pedagogy/instructional materials 


4 


Emphasized content somewhat more than pedagogy/instructional materials 


7 


Emphasized content and pedagogy /instructional materials about equally 


39 


Emphasized pedagogy/instructional materials somewhat more than content 


43 


Emphasized pedagogy/instructional materials far more than content 


6 



Table 7 

Projects’ Design for Professional Development: 



Purpose of Professiona 


Development 








Percent of Professional Development Hours 
(N = 70) 




Minimum 


Maximum 


Mean 


Standard 

Deviation 


Deepening teachers' knowledge of mathematics/science content 


10 


70 


29.57 


10.96 


Deepening teachers' knowledge of effective pedagogy 


10 


60 


31.43 


9.64 


Increasing teachers’ understanding of instructional materials to be 
used in the classroom 


5 


75 


33.07 


14.38 


All other purposes 


0 


25 


5.93 


5.79 



Evaluators’ narrative descriptions also suggest that LSCs emphasize pedagogy and materials, and 
with few exceptions, the designated instructional materials are the “centerpiece” of professional 
development efforts. First and foremost, professional development sessions provided 
opportunities for exploring the materials that teachers will use in their classes. Said this 
evaluator about this approach: 

Helping teachers become familiar with the modules designated for their grade levels is 
the predominant emphasis in the project ’s professional development design. Those 
activities that do not focus explicitly on learning to use the kits typically feature kit 
implementation as a vehicle for achieving other ( content or pedagogy-related) purposes. 
This is not a negative feature. The prevailing culture in most of the target schools does 
not give high priority to deep reflection and discourse on teachers ’ beliefs, knowledge, 
and practices. . .By framing professional development around kit implementation, the 
project is taking advantage of an existing ‘'hook” to attract participation. .. [and] to use 
the implementation as a context for examining content and pedagogy. (Evaluator, K-8 
science LSC) 

The majority of professional development around the designated instructional materials was 
delivered prior to teachers’ implementation of those materials. (See Table 8.) Given teachers’ 
immediate concerns about their ability to use the modules/kits in their classes, the level of 
emphasis on materials prior to implementation hardly seems surprising. Teachers typically had 
opportunities to explore the materials during summer workshops, and delve into the materials in 
greater depth or explore new units in sessions during the year. 
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Table 8 

Projects’ Design for Professional Development: Timing of Professional 



Development Relative to Implementation o 



Designated Instructional Materials 





Percent of Professional Development Hours 
(N = 70) 




Minimum 


Maximum 


Mean 


Standard 

Deviation 


Introducing teachers to the units/modules/kits and preparing 
teachers to implement the units/modules/kits for the first time 


0 


90 


54.07 


19.19 


Providing support during the time that teachers are implementing 
the units/modules/kits for the first time 


0 


50 


24.50 


13.08 


Reflecting on and refining implementation of the 

units/modules/kits after implementation for the first time 


0 


100 


21.43 


16.90 



Although LSCs were supporting the implementation of numerous modules/kits, 60 percent of 
projects tended to focus their professional development on particular units, while 40 percent 
attempted to spend equal time on each. (See Table 9.) However, in terms of activities within a 
unit/module/kit, 90 percent of projects focused on some activities more than others. 



Table 9 



Projects’ Design for Professional Development: 
Coverage of the District-Designated Instructional Materials 





Percent of Projects 






(N = 70) 




Units/modules/kits 


Activities within a unit/module/kit 


About the same amount of time spent on each 


40 


10 


Deliberately spend more time on some 


60 


90 



Evaluators reported that, on the whole, LSCs tried to integrate content throughout the 
design — through summer workshops and school-year follow-up activities. Courses at local 
colleges and universities, topic- specific workshops, seminars and symposia, and after-school and 
Saturday sessions were among the strategies used by LSCs to address teachers’ content needs. 
LSCs typically opted for the “logistical progression” in addressing content — first, helping 
teachers become comfortable with module/kit activities, and then focusing on deepening 
teachers’ understanding of key concepts through more advanced sessions. 

Evaluators consistently noted that LSCs placed a high level of emphasis on module-specific 
content in summer institutes. When professional development focused on mathematics/science 
content, it tended to be at the level that students were expected to learn. As shown in Table 10, 
on average, two-thirds of professional development hours were devoted to content at the student 
level, while one-third of the time was devoted to content beyond the student level. 
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Table 10 



Projects’ Design for Professional Development: 
Targeted Level of Mathematics/Science Content 





Percent of Professional Development Hours 
(N = 70) 


Minimum 


Maximum 


Mean 


Standard 

Deviation 


Content that students would be expected to learn 
Content beyond what students would be expected to learn 


10 

0 




65.71 

34.29 





In multi-district projects that supported more than one set of instructional materials, LSCs were 
more likely to examine content in light of standards, frameworks, and student learning outcomes, 
using content experts to increase opportunities for teachers to “make conceptual connections.” 
Projects also used workshops focused on assessment and student work to better integrate content 
across professional development opportunities; for example, assessment institutes in one LSC 
helped teachers make connections between key concepts and appropriate ways to determine 
students’ level of understanding. In some cases, LSCs increased the focus on content over the 
course of the project, altering introductory institutes to incorporate more content matched to state 
and national standards; developing mini-courses; and using videos in small group sessions for in- 
depth content analysis. Some university-designed courses went well beyond the student 
materials to the in-depth study of particular disciplines. 

Content sessions were sometimes designed in direct response to teachers’ demonstrated needs. 
For example, in a K-8 mathematics project, staff realized that teachers were having difficulty 
identifying the algebra embedded in the mathematics materials, and responded by adding an 
“algebra strand” workshop and conducting post-session interviews to help gauge the usefulness 
of the sessions for meeting teachers’ content needs. A few evaluators described the use of 
teacher performance assessments and pre-/post- workshop instruments to track teachers’ content 
needs, while other projects typically relied on classroom observations by LSC staff, teacher 
leaders, and principals, and data from workshop evaluations, school year surveys, interviews, 
focus groups, and teacher surveys. 

Strategies for Supporting Teachers 

LSCs differ from traditional professional programs in their attention to creating a year-round 
support structure for teachers. While summer sessions provided intensive opportunities for 
participants to explore new materials and practices over a period of one or more weeks, academic 
year follow-up sessions offered teachers the chance to examine relevant topics in depth, as they 
were engaged in implementation. Other areas of support provided by LSCs included materials 
management/refurbishment in science projects; school-based activities to promote ongoing 
discussion and reflection, and individualized support through coaching and mentoring to address 
specific needs. 

Small group activities designed to provide ongoing support for teachers included cross-grade and 
grade-level seminars, study groups, kit clubs, action research teams, “reflecting on teaching” 
groups, and other ki nds of structured “learning communities.” It was amply clear from teachers’ 
comments that participants appreciated the chance for this kind of support, but LSCs encountered 
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numerous barriers to providing these kinds of opportunities — teachers’ lack of time, scheduling 
conflicts, and so on. Some LSCs looked for ways to enhance the effectiveness of these small 
groups. For example, an urban LSC increased the number of university faculty used as 
mathematics “consultants” for school-based study groups to better support targeted teachers, 
while another LSC refocused the roles of mathematics lead teachers to work more with groups of 
teachers (as opposed to individuals) around planning and implementation. 

Other LSCs shifted their design to site-based leadership development, stepping up efforts in the 
final years to better prepare teacher leaders for facilitating and sustaining learning communities. 
Still, across LSCs, fewer than one quarter of LSC professional development hours were spent 
providing teachers support as they implemented the materials, or providing time for teachers to 
reflect on their initial implementation of the materials. Similarly, while Pis, teachers, and 
evaluators noted that coaching and/or mentoring were crucial for helping teachers to make 
meaningful changes in their practices, as noted earlier, LSCs’ devoted comparatively little time 
to one-on-one support for teachers, due in part to limited resources and staffing for engaging in 
these kinds of activities. 



Strategies for Garnering Support for the Institutionalization of LSC Reforms 

Improving student learning through teacher enhancement is a major goal of the LSC program. 
However, leaders of LSC projects are also expected to build stakeholder and policy support for 
mathematics and science education reform. Pis and evaluators alike spoke of the importance of 
engaging those both inside and outside the school community in support of reform, and 
establishing a presence for the LSC vision in the policy arena. 

Building Stakeholder Support 

School and district administrators were typically the primary targets for LSC efforts to garner 
stakeholder support. Projects used a variety of strategies for engaging school administrators. 
Some offered annual workshops for principals — ranging from several hours to several days — to 
brief administrators on goals, vision, and activities. Other LSCs created formal structures to 
involve principals on a more frequent basis. For example, in one LSC an administrative council 
provided the vehicle for principals to interact monthly with teacher leaders and project staff. 
Other projects deliberately hooked into existing structures for providing professional 
development for principals — for example, monthly meetings and networks already engaged in 
working with principals. One LSC tapped into the California Science Implementation Network, 
where principals and assistant principals participated on school leadership teams and in training 
along with their teachers. 

Other LSCs also took a highly proactive approach to building support among principals. In one 
urban LSC, project leaders devised a targeted and intensive strategy to gain principal support, 
conducting one and a half day retreats twice a year that included both teacher leaders and 
principals, and a series of three-day summer institutes for principals. Another LSC planned a 
Principals’ Forum to provide professional development for principals, including bi-monthly 
seminars for principals. Some LSCs worked intensively with a small group of principals to 
cultivate their support and skills for working with their colleagues; several LSCs added 
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Principals-in-Residence for this same purpose — released full time from their school 
responsibilities to work with other administrators. Other LSCs took principals to national level 
conferences and professional development to broaden their understanding of the reform vision, 
and brought in national experts to work with principals locally. Still others looked for ways to 
align the LSC vision with principal “routines,” for example, teacher evaluation, and used the 
HRI Classroom Observation Protocol and LSC-developed instruments to educate principals in 
this area. Where it was difficult to convene principals, LSC staff worked one-on-one with school 
administrators, or counted on lead teachers to engage principals. 

LSCs used similar kinds of strategies to engage district administrators, working through existing 
channels and through one-on-one outreach. Some LSCs collaboratively planned and advertised 
professional development with central office staff, and coordinated with central office 
departments — staff development, literacy, bilingual, and special education — to increase 
awareness and support. In an elementary science LSC, for example, project staff held monthly 
meetings with elementary education directors from the four participating districts, with 
discussions focused on strategic planning, professional development, and district policies to 
support reform. Another multi-district LSC facilitated cross-district meetings with 
superintendents to build support for project strategies. Project staff and teacher leaders also 
participated on district committees for frameworks, curriculum, assessment, and professional 
development to give voice to the LSC vision among central office staff. As with principals, 

LSCs also recruited district administrators to attend national level leadership development 
sessions to build their understanding of the vision, goals, and activities needed for building 
inquiry-based mathematics/science programs. 

Typically, parent outreach efforts occurred at the school-level, as opposed to project-wide, 
especially in LSCs targeting multiple districts spread over large geographical areas. 

“Awareness” workshops, leadership development seminars for parents, Family Math and Science 
Nights, and dissemination of printed materials about the LSC were among the strategies used by 
LSCs to gain parent support. Some LSCs also looked for more creative ways to engage parents 
as partners in reform — for example, as volunteer “associates” to assist with materials 
management. Pis of mathematics LSCs sometimes reported using a “cautious” approach in 
working with parents and the community, due to conflicting agendas around mathematics 
instruction. For example, one project used a highly inclusive, community-oriented process for 
reviewing instructional materials. Another used an outside consultant, organized small groups of 
parents in the evenings, offered childcare to encourage attendance, and used business volunteers 
to help. 

Efforts to secure teacher union support occurred primarily through the work of teachers who 
played leadership roles in both their unions and in the LSC, and who sought buy-in and 
communicated vision and progress through their union involvement. It appeared from PI 
interviews, however, that few LSCs had a proactive strategy for building union support. Rather, 
strong word-of-mouth “advertising” by teachers, as well as shared data on improved student 
achievement, appeared to be the key strategies for fostering teacher union support. 

In developing partnerships to support reform, LSCs reported communicating their vision through 
meetings with university faculty and administrators, hiring liaisons to facilitate partnerships 



Horizon Research, Inc. 



22 



December 2006 




between the university and the district(s), and working with college/university education 
departments to ensure that student teachers were exposed to district-designated instructional 
materials and student teaching experiences in LSC schools. Some projects actively sought 
business support, giving presentations to local businesses, and sponsoring school-based and 
project- wide activities to increase awareness and show appreciation for support. Other LSCs 
developed collaborative community efforts to engage local scientists and engineers. 

Strategies for Building Supportive Policies 

LSCs were frequently able to “ride the wave” of policy support initiated by national and state 
trends toward standards-based instruction. Using this policy-friendly environment to their 
advantage, LSCs promoted the selection of materials aligned with district, state, and national 
standards, and supported districts in their adoptions. Typically, LSCs worked to achieve a strong 
presence of reform leaders on curriculum committees charged with reviewing materials and 
writing or revising local frameworks. For example, LSC leaders in one project focused on 
curriculum policies at the school and district levels to influence the decision-making process, 
working with both teachers and administrators and sending some to a regional workshop on the 
selection of new curricula. In another LSC, the entire design was based on a strategy for dealing 
with policies that emphasized literacy development; it was a design specifically aimed at 
building a niche for science instruction within a policy environment where literacy was the 
primary driver for instruction. 

Other LSCs took a highly proactive approach to influencing curriculum policy. In one project, 
LSC leaders developed an instructional analysis tool to help schools select curricula through an 
extensive review process. To encourage articulation across grade levels, the LSC designed a 
two-day seminar to showcase materials that best aligned with the exemplary high school 
materials already being implemented. The project also created a “framework” for reform that 
made explicit areas of alignment in the materials and in the curriculum guidelines. Finally, at the 
time the LSC was funded, the state was moving toward a licensure program for teachers, and 
project leaders took these requirements into consideration as they were developing professional 
development for middle school teachers. Many other LSCs actively worked with 
superintendents and others, taking advantage of the timing of the adoption process to promote 
high-quality instructional materials and ensure that frameworks developed with LSC guidance 
would be adopted. In some cases, LSCs also worked with district personnel to submit waivers to 
allow the use of modules from several different programs, some of which were not on the state- 
approved list. 

While some LSCs deliberately took on the work of developing policy support, others chose a 
more indirect, “grassroots” approach by relying on the development of stakeholder support. Said 
one PI of this approach: 

I think the original idea was to empower teachers to influence district curriculum policy 
and make what works with kids and teachers at the grassroots level more influential in 
the decision making process. (PI, K-8 science LSC) 

Similarly, in another LSC where state-level policies conflicted with the LSC vision, project 
leaders sought to build capacity and provide tools for local decision-makers to function in this 
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increasingly hostile policy environment. The strategy included working with lead teachers, who 
became advocates for using science to build language skills; with LSC support, lead teachers 
developed district curriculum guidelines, reviewed instructional materials, and took their 
recommendations to school boards and curriculum committees. The result was the development 
of local science content matrices aligned with the LSC vision, and the adoption of aligned 
materials in the districts to supplement state-sanctioned texts. 
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Part Three 



Quality and Reach of LSC Professional Development 



The success of an LSC depends in part on the quality of the interventions and the ability of the 
project to engage the targeted audience in these activities. As described in the previous section, 
LSC projects varied widely in their strategies for translating the LSC theory of action into a set 
of interventions. Whatever the particular design, however, the theory argues that LSCs needed 
to include a number of dimensions in their professional development efforts: well-prepared 
professional development providers; a supportive and collegial learning environment for 
teachers; opportunities for teachers to enhance knowledge around content, pedagogy, and high- 
quality instructional materials; and ongoing support for teachers as they implement the materials. 

Balancing attention to the various dimensions is critical for success. Typically, however, LSCs 
were faced with trade-offs within and across these dimensions that may have affected the quality 
of the professional development and the numbers of teachers that participated. This section 
looks at the extent to which LSCs reached eligible teachers, and some of the factors that 
influenced both the quality and reach of LSC professional development. 



Reaching Teachers 

Unlike previous teacher enhancement initiatives which focused their efforts on “volunteers,” 
LSCs were expected to provide all teachers in targeted schools or districts with 130 hours of high 
quality professional development. Different projects approached this task with different 
strategies to reach their targeted audiences. For example, some worked with cohorts of schools; 
others worked first with “volunteers” — teachers who were more willing and ready to enlist in 
reform-based activities; still others focused first on teacher leaders before engaging “typical” 
teachers. 

The strategy of trying to reach all teachers can be seen as either a success or a failure depending 
on the criterion applied. Across projects, evaluators noted that LSCs expended considerable 
effort to provide opportunities for accessible and classroom-relevant professional development in 
order to reach the 130-hour goal. As a result, LSC projects provided a great deal of high quality 
professional development to a very large number of teachers, almost certainly many more than 
would have been reached with comparable funding if the goal was set lower than whole 
schools/districts. Still, providing the required number of professional development hours to all 
eligible teachers challenged the majority of LSCs. Although a greater proportion of teachers 
received professional development as projects matured, no project accomplished the goal of 
providing 130 hours of professional development to all targeted teachers, and relatively few 
teachers across LSCs reached this goal by the end of the project. (See Table 11.) 
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Table 11 



Teacher Report of Participation in LSC Professional Development, 
by Data Collection Year 





Percent of Teachers 


Overall 


Baseline Year 


Year Two 


Final Year 


0 hours 


35 


81 


34 


21 


1-19 hours 


20 


11 


23 


18 


20-39 hours 


12 


4 


14 


13 


40-59 hours 


10 


2 


10 


12 


60-79 hours 


6 


1 


6 


7 


80-99 hours 


5 


0 


4 


7 


100-129 hours 


6 


0 


5 


10 


130 or more hours 


6 


1 


5 


11 



Teacher turnover was one reason that so few projects reached their goal in this area. References 
to teacher turnover are extensive in evaluator reports, with many evaluators citing turnover as a 
leading barrier to providing the expected number of professional development hours to eligible 
teachers. Mobility across schools and subject areas, retirement, resignation, and staff downsizing 
all contributed to the problem. Evaluators suggested that LSCs may have sorely underestimated 
or ignored teacher turnover in designing their interventions, and were left unprepared for the 
consequences. The influx of new teachers had huge implications for delivering professional 
development to all targeted teachers: LSCs typically had to reach many more individuals than 
they planned to and had difficulty doing so. In California, in particular, mandated decreases in 
class sizes led to the hiring of large numbers of additional teachers. Original projections of 
eligible teachers became “moving targets” as LSCs sought to treat new teachers, and provide 
ongoing professional development and support to initial participants. 

Some schools lost teachers at higher rates than others, leaving participants without collegial 
support, encouragement, or “peer pressure” to continue with their participation. An analysis of 
teacher sampling frames submitted by projects in their Linal Year indicates that, across the LSC 
program, projects lost an average of 13 percent of their targeted teachers each year. Roughly 70 
percent of the teachers leaving LSC districts had received some professional development; one 
quarter received 60 or more hours of LSC professional development. 

Other barriers faced by LSCs may also shed light on the challenge of meeting the 130-hour goal. 
Losses in state- and/or district-supported professional days that had previously supported LSC 
activities sometimes decreased opportunities for participation. Where districts did not adopt the 
designated instructional materials, or where high stakes assessments dominated the instructional 
landscape, principals were less apt to support participation, and teachers felt less inclined to 
attend LSC professional development that they perceived as irrelevant and contradictory to 
district goals. Similarly, when changes in school, district, and state initiatives called for new 
professional development priorities, teachers were sometimes less inclined to participate in LSC 
activities. 

Evaluators reported that LSCs, faced with these barriers, tried to modify their designs to better 
reach targeted teachers. Lor example, when the lack of substitutes plagued LSCs’ efforts to 
conduct professional development during the school day, projects tried to modify professional 
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development schedules, but participation typically diminished when activities occurred on 
Saturdays or after school. Some projects had more success than others in making adjustments to 
better reach teachers. In one project, for example, the evaluator attributed high and steady rates 
of participation to the LSC’s extensive and sustained recruitment efforts — through meetings, 
presentations, and outreach efforts to principals and faculty beginning early in the school year. 

Other LSCs presented their vision in light of state standards, assessments, and competing 
priorities in order to address teachers’ immediate concerns in these areas and sustain their 
participation. In one project, for example, the LSC linked student success in English language 
acquisition with science instruction, thereby enabling teachers to address the state and district 
emphasis on literacy while also improving science instruction; as a result, teachers perceived the 
LSC reforms as highly relevant, and summer sessions were well-attended. LSCs also provided 
incentives for participation, including college credit, stipends, and salary increment credit. They 
also linked LSC professional development to other priority areas, typically literacy, but also 
technology. Still, without a “mandate” for participation — which few projects were able to 
provide — LSCs had to rely on teachers’ own motivation; the sanction of schools, districts, 
teacher unions, and others to support participation; and project leaders’ capacity to make the case 
for participation. 

Teacher attitudes about district-designated instructional materials and instructional strategies 
promoted by the LSC, and about their perceived needs in these areas, may also have deterred 
participation in some cases. Lor example, instilling the notion of ongoing professional growth 
posed challenges: according to some evaluators, once teachers had used the materials in their 
classrooms and believed they understood inquiry-based instruction, they saw little need to 
participate further. In many cases, sustaining teacher involvement after the initial summer 
activities was difficult. Evaluators reported that LSCs encountered particular problems in 
engaging secondary level teachers who were apt to be more critical of LSC pedagogical 
approaches, and to feel less of a need for enhancing their content knowledge. Lor elementary 
teachers, workload and the number of subjects taught may have reduced the priority to 
participate in LSC professional development, particularly sessions on content beyond what 
teachers perceived was needed to implement the kits. 

While these explanations may account for some of the difficulties LSCs encountered in reaching 
targeted teachers, there is some evidence to suggest that project strategies used for engaging 
teachers also contributed to lower participation rates than expected. LSCs that used a phase-in 
approach beginning with “volunteers,” left more resistant teachers to later years in the project. 
The strategy, while assuring enthusiastic “pioneers” to get the reform going, caused its own set 
of problems down the road. Resistors who balked at participation, and at the proposed reforms 
in general, demanded greater energy from project staff, which in turn diminished ongoing 
professional development opportunities and support for earlier cohorts of teachers. 

LSCs that worked with schools where administrative support was lacking, where the school 
culture was not conducive to reform and collaboration, or where there was community-based 
opposition to reform approaches seemed particularly susceptible to this kind of resistance and, 
consequently, decreased participation. In cases where the district context changed over the 
course of the LSC, project leaders were forced to make difficult choices about the targeted 
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audience. For example, when policies called for a reduction in class size and the hiring of 
hundreds of new teachers, one LSC decided to target the newly hired teachers, many of whom 
were inexperienced and operating with an emergency credential. Project leaders believed that 
introducing these teachers to high-quality materials and pedagogy in science would have a “long- 
term pay-off’ by ensuring that new hires would “start their careers by thinking that science is an 
essential part of the curriculum.” By concentrating on new teachers, however, the project 
reached only 15 percent of the teaching population in the district, falling far short of the original 
LSC goal. ' 

Finally, inaccuracies in teacher self-report data on participation may have contributed to the 
underestimation of time spent in LSC professional development. For example, evaluators 
suggested that teachers might not have always known whether a professional development 
session was provided by the LSC. As one evaluator reported: 

Teachers had difficulty identifying the project by name, differentiating it from other 
mathematics-related programs within the schools, [which led to] underestimating their 
participation as documented through project attendance records. (Evaluator, 6-12 
mathematics LSC) 

Other evaluators suggested that teachers might not have attributed school-based professional 
development activities such as coaching and mentoring to the LSC, thus reducing the number of 
hours of participation they reported. A comparison of teacher-report of participation in LSC 
professional development on the teacher questionnaire and project records in the teacher 
sampling frame supports the hypothesis that teachers underestimated their participation in LSC 
professional development. (See Table 12.) Still, even by project reports, only 14 percent of 
teachers received the targeted 130 hours of professional development. 



Table 12 



Teacher Participation in LSC Professional Development, 
Teacher- Report vs. Project- Report for Final Year Projects 





Percent of Eligible Teachers 


Teacher-Report 


Project-Report 


0 hours 


21 


17 


1-19 hours 


18 


13 


20-59 hours 


24 


24 


60-99 hours 


14 


19 


100-129 hours 


10 


14 


130 or more hours 


11 


14 



Quality of LSC Professional Development Programs 

The evaluator of each LSC project was asked to rate the general quality of that project’s 
professional development program — both overall and on a number of dimensions. A regression 
analysis (Shimkus & Banilower, 2004a) using hierarchical modeling, with time points nested 
within projects, indicates that evaluator ratings of the overall quality of the professional 
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development program increase significantly over time. (See Table 13.) The same pattern was 
found on 4 of the 5 dimensions on which projects were rated: culture of the professional 
development program; quality of preparation of the professional development providers; helping 
teachers become familiar with the instructional materials/pedagogy; and deepening teacher 
mathematics/science content preparedness. 5 



Table 13 



Predicted Probabilities of Ratings r of the LSC Professional Development Program, 
by Project Year 





Predicted Probability 




Baseline 


Year 


Year 




Year 


Two 


Five 


Overall program rating* 

Low 


0.02 


0.01 


0.00 


Medium 


0.79 


0.54 


0.14 


High 


0.17 


0.42 


0.64 


Very High 


0.01 


0.04 


0.23 


Culture of the professional development program* 

Low 


0.01 


0.00 


0.00 


Medium 


0.27 


0.14 


0.04 


High 


0.61 


0.63 


0.44 


Very High 


0.11 


0.22 


0.52 


Quality of preparation of professional development providers* 

Low 


0.02 


0.01 


0.00 


Medium 


0.29 


0.13 


0.03 


High 


0.61 


0.66 


0.44 


Very High 


0.08 


0.20 


0.53 


Helping teachers become familiar with the designated instructional materials 








and learn the appropriate pedagogy to develop students’ conceptual 
understanding of mathematics/science* 

Low 


0.03 


0.01 


0.00 


Medium 


0.33 


0.18 


0.06 


High 


0.60 


0.72 


0.69 


Very High 


0.04 


0.08 


0.24 


Deepening teachers’ understanding of mathematics/science content* 

Low 


0.03 


0.02 


0.01 


Medium 


0.55 


0.43 


0.26 


High 


0.39 


0.50 


0.62 


Very High 


0.03 


0.05 


0.11 


Supporting teachers as they implement the designated instructional materials 
in their classroom 








Low 


0.02 


0.02 


0.01 


Medium 


0.39 


0.35 


0.30 


High 


0.48 


0.51 


0.54 


Very High 


0.10 


0.12 


0.15 



' Evaluators rated programs on a five-point scale, where 1 and 2 were considered low ratings, 3 a medium rating, 4 a high 



rating, and 5 a very high rating. 

* Evaluator ratings increase significantly as projects mature, p < 0.05 



5 Although the ratings for providing support to teachers as they implement the designated instructional materials 
appear to increase over time, this pattern is not statistically significant. 
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Of the five dimensions of LSC professional development programs, the data suggest that LSCs 
had particular success in creating a professional development culture conducive to teacher 
learning; 96 percent of the projects received a rating of 4 or 5 in the Final Year. Based on 
evaluators’ ratings, LSCs appeared to be highly adept at creating professional development 
programs in which trust, risk-taking, and collaboration prevailed. Using current or former 
teachers as professional development providers likely contributed to this achievement. Evaluator 
ratings also suggest that LSC programs did well in preparing professional development 
providers, with 97 percent of the projects receiving a “High” or “Very High” rating in the final 
project year. 

LSCs were designed to help teachers become familiar with the designated instructional materials 
and leam the appropriate pedagogy to develop students’ conceptual understanding of 
mathematics/science. Evaluators rated LSC programs highly in this category, with 93 percent of 
projects receiving a rating of 4 or 5 in their Linal Year. Only 73 percent of LSC programs 
received high ratings in their efforts to deepen teachers’ understanding of mathematics/science 
content. 

The ratings for providing support to teachers as they implement the designated instructional 
materials suggest that LSCs also had some difficulty in implementing designs for providing 
ongoing support. Evaluators suggested that the limited number of qualified teacher 
leaders — who were typically responsible for providing this support — often hampered the efforts 
in this area. On the positive side, teachers consistently cited the availability of materials and 
supplies as one of the most helpful aspects of LSC programs. Hands-on instruction requires 
considerably more attention to consumable materials than does a textbook-oriented approach; 
according to evaluator reports, LSCs were highly responsive to teachers’ needs in this area, 
“making it easy” for teachers by maintaining, restocking, and delivering materials, and keeping 
accurate records on dispersal and usage. In the case of some elementary science LSCs, 
evaluators attributed teachers’ high level of commitment, and the likelihood of sustainability, to 
an efficient and well-financed materials management system. 

In some cases, LSCs were able to enhance their overall program in ways that resulted in a solid 
system of professional development and support for teachers, as described by this evaluator: 

Especially when comparing the status of science education in the target region a mere 
four years ago, the progress that the project has achieved is great and has occurred on 
multiple fronts. Today a strong and comprehensive professional development program 
exists. It has a well-organized system of mandatory workshop offerings that is staffed by 
a cadre of increasingly well trained, capable and experienced teacher leaders. The 
materials and supplies support system is efficient and responsive, housed in an expanded 
Mathematics/Science Resource Center that serves as an important hub of activity and 
supports teachers with the fundamentals necessary for hands-on science instruction. 

There is also the [Science Resource Teacher] component which incorporates both in- 
classroom support to individual teachers, as well as school-level workshops tailored to 
individual school needs. Finally, there is a very good selection of second tier 
professional development activities designed for teachers who are ready to deepen their 
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repertoire. The selection includes a wide array of content specific institutes, facilitated 
by carefully orchestrated teams of scientists, [Science Resource Teachers], and teacher 
leaders, as well as those focused on pedagogical issues and strategies. (Evaluator, K-8 
science LSC) 

In this case and others, evaluators described the ways in which project leaders moved beyond the 
early focus on “basics” (kit trainings and materials management) to more challenging 
programmatic design issues, including strengthening the skills of teacher leaders, developing a 
community of learners among LSC professional development providers, and providing 
individual and school support tailored to needs. 

With multiple roles in LSC projects, teacher leaders’ time was also severely restricted, thus 
hampering their ability to provide the degree of support requested and needed by teachers during 
implementation. When LSCs added sessions to support new teachers, these efforts stretched 
professional development providers even further and diminished their ability to provide the 
necessary support for other teachers. The lack of “structures” for support also deterred these 
kinds of opportunities for teachers, as these evaluators note: 

In general, the strength of the [LSC] project was to provide adequate professional 
development so that a portion of [district] elementary and middle school mathematics 
teachers implemented quality inquiry-based curriculum resources. Although the ongoing 
support during the academic year appears to have helped teachers while implementing 
the materials, individualized support has not yet been sufficient to promote most teachers 
to high levels of implementation. Organizational structures that support reflection about 
teaching practice and lead teachers to improve the quality of instruction were needed. 
These structures might also provide more opportunities for teachers to continue 
improving their mathematics content knowledge. (Evaluator, K-8 mathematics LSC ) 

Overall, the [LSC] project continued to make improvements in its design and 
implementation. Project leaders have been flexible in recognizing and responding to 
challenges, particularly concerning reaching reluctant teachers. The project ’s 
professional development and materials distribution have enabled teachers to use the kits 
confidently and effectively, and the quality of the project ’s attention to pedagogy and 
content continues to improve. However, the project’s overall effectiveness remains 
challenged by limitations in its system for participants’ ongoing professional 
development and support during the school year. (Evaluator, K-8 science LSC) 

Opportunities for teachers to receive support as they teach their designated materials, 
while available in the broadest sense, have ostensibly disappeared. Teachers feel 
supported as they are trained, and have been invited informally to call upon mentor 
assistances and facilitators as they teach, but that there are no specific structures for 
ongoing mentoring or coaching is a weakness of the program. The project has thus come 
to rely on building level support for teachers. While this is an important mechanism to 
develop across a district of this size, it is also difficult to maintain in an environment 
where there have been so many staff changes, and it is often difficult to implement in any 
formalized manner. (Evaluator, K-8 science LSC) 
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Some Pis and evaluators also noted the value of coaches, some of whom were “at the core” of 
what made the LSCs successful. Said this evaluator: 



When looking across classroom observation data, those teachers who effectively 
implement the curriculum are the ones who have received more one-on-one, in-depth 
professional development provided through the cohort school structure. (Evaluator, K-8 
science LSC ) 

While some LSCs were attentive to the preparation of teacher leaders for coaching roles — for 
example, providing sessions on the Cognitive Coaching model, and developing “planning and 
reflecting maps” to help in observing and engaging teachers in dialogue — others were less 
deliberate in these efforts. Evaluators frequently noted that coaches/mentors lacked the capacity 
to adequately support teachers using these strategies. Attrition and turnover sometimes resulted 
in less experienced coaches — with limited training and undeveloped leadership skills. These 
shortcomings likely affected the quality of support sessions, which evaluators sometimes 
described as “overly mechanical” or “superficial,” as opposed to well-crafted professional 
development opportunities. 

The demand for logistical support also may have decreased the impact of these types of sessions. 
For example, teachers noted that coaches provided “an extra set of hands,” reordered materials, 
and kept everyone up to date and “on track.” While this kind of support was crucial for helping 
teachers use the modules, it also likely diminished the quality of one-on-one sessions intended 
for deepening content or pedagogical understanding. 



Quality of LSC Professional Development Sessions 

Although evaluators typically assigned high ratings to LSC professional development sessions, 
there was also ample evidence to suggest that LSCs encountered some challenges in this area. 

For example, projects struggled to find a balance between content, pedagogy, and materials in 
professional development sessions. The dynamic tension between depth and breadth of coverage 
also posed difficulties. LSCs sometimes struggled in their efforts to provide sessions that moved 
teachers beyond module- specific discussions to deeper conversations around conceptual 
understanding and student learning. Finally, sometimes the design of a session was appropriate 
but the flawed implementation failed to deliver the quality of intervention that was intended. 

As part of the core evaluation, LSC evaluators were required to observe a sample of professional 
development sessions each year of the project and to assess the quality of these individual 
sessions. The sampled sessions, although not randomly selected, were intended to be 
representative of the professional development offered by each project. As shown in Table 14, 
session ratings in the projects’ baseline year were roughly split between medium and high 
quality, while in subsequent years the majority of sessions received high ratings. A longitudinal 
regression analysis conducted by Banilower and Shimkus (2004) using hierarchical modeling 
indicates that LSC professional development sessions were more likely to be rated highly as 
projects matured. 
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Table 14 

Predicted Rating of Session Quality, ' * 
by Project Year 





Predicted Probability 


Baseline Year 


Year Two 


Year Five 


Low 


0.09 


0.06 


0.03 


Medium 


0.44 


0.36 


0.24 


High 


0.47 


0.58 


0.72 



1 Evaluators rated sessions on a five-point scale, where 1 and 2 are considered low 



ratings, 3 a medium rating, and 4 and 5 high ratings. 

* Ratings of session quality increase significantly as projects mature, p < 0.05 



Table 15 shows that evaluator ratings of the quality of observed LSC professional development 
sessions were similar regardless of professional development purpose. 6 It is important to note 
that sessions could and often did have multiple purposes. The majority of sessions, regardless of 
purpose, received high ratings by evaluators. 

Table 15 



Ratings 1 of LSC Professional Development Session Quality, 
by Professional Development Purpose 









Percent of Sessions 
(standard error) 






N 




Low 


Medium 


High 


Promoting/exploring reflective practice 

Understanding student thinking/learning about mathematics/science 


440 


4 


(1.1) 


27 


(2.5) 


69 


(2.6) 


content 


640 


6 


(1.1) 


27 


(2.0) 


67 


(2.1) 


Creating a vision of effective mathematics/science instruction 
Explicit attention to strategies/issues/roles of teacher leaders, principals, 


652 


6 


(1.1) 


28 


(2.1) 


66 


(2.2) 


or others in leadership positions 


304 


6 


(1.6) 


28 


(3.1) 


66 


(3.2) 


Building professional networks among educators 


307 


6 


(1.6) 


31 


(3.1) 


64 


(3.2) 


Increasing mathematics/science content knowledge of participants 


893 


7 


(1.0) 


30 


(1.8) 


63 


(1.9) 


Learning pedagogical/classroom management strategies 


707 


7 


(1.1) 


32 


(2.1) 


61 


(2.2) 


Learning how to use specific instructional materials in the classroom 


952 


8 


(1.1) 


35 


(1.8) 


56 


(1.9) 



' Evaluators rated sessions on a five-point scale, where 1 and 2 are considered low ratings, 3 a medium rating, and 4 and 5 



high ratings. 



6 Using the standard errors, it is possible to judge whether the difference between two estimates are statistically 
significant. Generally, if the observed difference between two estimates is at least twice the standard error of the 

I 2 2 

difference, where SE dj g = y SE X + SE^ , the two estimates are statistically different. This method of determining 

statistical significance is conservative for two reasons. First, the standard error of the difference will be smaller if 
there is covariance between the items. Second, the estimate is typically used for independent samples; since 
professional development sessions could include more than one purpose, the observation data are not entirely 
independent, and therefore there is likely a greater amount of covariance among the items. 
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Banilower and Shimkus (2004) also examined the relationship between session quality and a 
number of session characteristics, including the type of professional development provider and 
the type of professional development activity. The study identified several factors that seem to 
influence session quality. First, engaging participants in problem-solving/investigations appears 
to be an effective approach for a number of professional development purposes, including 
creating a vision of effective instruction, learning to use instructional materials, building 
professional networks, learning classroom management/pedagogical strategies, and 
understanding student thinking. A general principle that may underlie these findings is that 
active learning opportunities and teacher-to-teacher interactions are attributes of high quality 
professional development (Garet, Porter, Desimone, Birman, & Yoon, 2001). These attributes 
seem more likely to characterize sessions that engage teachers in problem-solving and 
investigations. 

The Banilower and Shimkus study also found that utilizing district-designated instructional 
materials in professional development appears to be an effective strategy for deepening teacher 
content knowledge and preparing teachers to implement those materials in the classroom. 
Professional development sessions focusing on the use of materials were likely to be “practice- 
based” and directly connected to the work that teachers do in their classrooms. Consequently, 
these sessions may be more engaging to teachers, and presumably more likely to translate into 
changes in instructional practice. 

As noted earlier, LSCs typically used a variety of professional development approaches, 
including workshops, study groups, classroom demonstrations, coaching, mentoring, and so on. 

A third pattern that emerged in the Banilower and Shimkus (2004) study was that the 
professional development approach (workshops versus study groups/coaching/mentoring) was 
not a significant predictor of session quality. In only two of the eight purposes of professional 
development — creating a vision of effective instruction and providing strategies in leadership 
roles — was quality linked to the type of approach. Interestingly, workshops tended to be more 
effective for the purpose of creating a vision of effective instruction, and less effective for 
providing strategies for leadership. 

Professional Development Providers and Session Quality 

In their study of LSC professional development observation data collected between 1998 and 
2003, Banilower and Shimkus (2004) looked at session quality in light of the type of professional 
development provider leading a session. Among the providers considered were regular 
classroom teachers; teachers on special assignment; district mathematics/science supervisors 
and/or other district personnel; college/university mathematics/science faculty and/or business 
industry scientists; college/university mathematics/science education faculty; and other non- 
district personnel (e.g., museum personnel or textbook representatives). For 5 of the 8 
professional development purposes examined, the type of provider was not an important 
predictor of session quality. The exceptions were sessions focused on creating a vision of 
effective instruction, promoting reflective practice, and understanding student thinking, which 
tended to be rated more highly when mathematics/science education faculty were among the 
facilitators. 
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Although evaluators gave projects high marks for the quality of preparation of professional 
development providers when completing the project ratings form, evidence from evaluator 
reports suggests that the skills, background, and preparation of professional development 
providers may have more influence on session quality than the type of provider. The more 
skilled facilitators were able to convey a vision for effective instruction; connect content, 
pedagogy, and materials; and create a professional development culture that supported rigorous 
investigation and inquiry. Evaluators also pointed to several areas where professional 
development providers struggled: presenting content at appropriate levels, conveying conceptual 
understanding beyond module-based content, and making content connections explicit. Again, 
LSC evaluators often attributed these shortcomings to under-prepared professional development 
providers. For example, some content experts presented topics that were inappropriate for the 
participants — beyond their backgrounds and/or without clear links to the student materials. 
Sessions facilitated by teacher leaders sometimes lacked rigor, particularly around content and 
student learning. This comment by an evaluator about the variable quality of sessions led by 
teacher leaders was fairly typical: 

There was considerable variation in the effectiveness of teacher leaders. Although all 
received essentially the same level of training and other support, some did better than 
others in facilitating PD for their colleagues. It was clear from evaluator conversations 
with these teachers that facilitating PD for adults is different than working with their 
students. Some were able to make the transition from teaching students to facilitating PD 
better than others. (Evaluator, K-8 mathematics LSC) 

Observations by evaluators suggest other weaknesses as well. Missed opportunities for 
reflection and deeper exploration of concepts led one evaluator to describe a session that seemed 
“more of a show than an integration of content, process, reflection, and application.” Others 
described discussions that were “limited in time and scope” or “surface-level.” Some 
professional development providers tried to cover too much material — for example, describing 
an entire module, lesson by lesson. These kinds of sessions typically seemed to leave teachers 
with a limited understanding of the relationship between inquiry, activities, and the underlying 
concepts. 

Varying levels of experience, as well as teacher leader turnover and reassignment, challenged 
LSCs in maintaining a base of well-prepared providers to lead professional development 
sessions. Yet strategies for ensuring quality control were typically scarce across LSCs. Said 
these evaluators about the lack of a systematic approach to preparing, supporting, and monitoring 
professional development providers: 

Concerns still remain regarding how the project identifies and addresses the differences 
in background among the project mentors. The training session focused principally on 
their kit familiarity and the logistics of the workshop session planning; again, because 
these are respected teachers in the region, it is assumed that they all have the 
background knowledge in content, pedagogy, and adult learning principles to implement 
their plan in a high-quality manner... Mentor training, as currently carried out by the 
project, does not identify and address the differing needs of the individuals in the mentor 
group. (Evaluator, K-8 science LSC) 
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What was missing throughout the previous two years was a systematic approach to 
articulating and communicating standards of effective professional development, and a 
mechanism or structure to assess and enhance teacher leader practices, and to provide 
opportunities for obseri’ing and reflecting on the quality of professional development 
practices implemented by teacher leaders in the context of the standards. (Evaluator, K- 
12 mathematics LSC) 

The [Leadership Team meetings] represent a significant effort on the part of project staff 
intended to develop in-house providers of professional development and thus to ensure 
the continuation of the project after LSC funding is gone. However, it is important to 
note that, in genercd, there is no systematic mechanism for determining the qucdifications 
of providers of professional development or for monitoring the quality of professional 
development being offered by the districts. (Evaluator, 6-12 mathematics LSC) 

Projects sometimes relied on co-facilitation of workshops, mentoring, and formative feedback 
from participants for ensuring quality. Typically, however, LSCs lacked the resources or staffing 
needed to monitor the work of providers, particularly in multi-district projects. 

Culture in LSC Professional Development Sessions 

Evaluators reported that professional development providers — particularly teachers on special 
assignment — were strong in creating sessions where a collegial atmosphere prevailed and where 
participants’ own knowledge and background were valued additions to the conversation. LSCs 
typically structured professional development sessions in ways that promoted dialogue and a 
joint sense of purpose — where all participants were encouraged to contribute. In larger group 
settings, staff developers created a seminar-like atmosphere in breakout sessions where 
participants collaborated as learners and reflective practitioners. 

At the same time, LSCs encountered some challenges in creating an atmosphere in which 
teachers felt at once comfortable and “stretched” in their knowledge and beliefs. Teacher leaders 
sometimes reverted to didactic approaches that limited participant interaction, problem-solving, 
and deeper exploration of topics. Said this evaluator: 

We have seen that the professional events do indeed offer designs which are intended to 
promote reflection and discussion, but we have not seen great evidence of animated, 
engaging conversation among teachers. Rather, in general, we have seen that teachers 
have had a difficult time conversing with a focus on instructional issues, and 
simultaneously, the professional development providers have been challenged to 
encourage these discussions. Thus, the implementation of the design has been 
problematic... [The project] has been very successful in establishing a culture where all 
are welcomed and included. However, now in its third year, we feel the next 
developmental step is to devise ways to create a more rigorous and intellectual culture, 
where teachers can learn to become more reflective practitioners. (Evaluator, K-8 
science LSC) 
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Scientists/mathematicians who led sessions ran a similar risk in creating a less than participatory 
experience for teachers. For example, in describing sessions facilitated by industry scientists, 
one evaluator noted, “Most often, the scientist talked and participants listened and responded.” 

In these cases, evaluators suggested that inadequate preparation of professional development 
providers — both teacher leaders and content experts — may have contributed to a less “open- 
ended” approach that reduced the investigative/problem-solving aspects of the sessions that 
appear to be linked to teacher engagement and higher quality. 

Quality of Content Sessions 

Program ratings for professional development around content were typically lower than for other 
professional development dimensions. Observations by evaluators revealed some weaknesses in 
individual content sessions that may help explain the lower overall ratings. Providing a 
“framework” for module-and kit-related content within the broader curriculum seemed to be a 
key challenge, as this evaluator notes: 

The project is strong in its support of the curriculum, quite strong in its support of 
pedagogy, and offers extensive opportunity for classroom support. Its primary limitation 
is an undeveloped vision of how to incorporate the big ideas of mathematics, and the 
importance of children ’s thinking into a conceptual framework that supports their work. 
...Although isolated mathematics activities separate from the curriculum have been 
included, the facilitators have not connected these activities to a large mathematical 
framework. (Evaluator, K-8 mathematics LSC) 

Other evaluators described sessions in which content was not included in “any systematic way”; 
where facilitators shifted the discussion to logistics and teaching tips instead of addressing 
teachers’ content questions; where facilitators were “unwilling” or unable to probe further 
around participants’ content understanding; and where providers neglected to reinforce the “big 
idea” of the session — for example, linking the mathematics content of individual sessions to the 
overall content of the professional development focused on algebraic thinking. Said these 
evaluators: 

To elevate the quality of the kit workshops still higher, the focus needs to expand to 
include an emphasis not only on the procedures and organization of the kits, but on the 
science content and the conceptual development embedded within them... The sole focus 
on the mechanics of the kits does not allow teachers to reflect on their teaching practices 
or the rationale for hands-on, inquiry-based science education. (Evaluator, K-8 science 
LSC) 

By focusing on the curriculum as the vehicle for covering content, teachers often feel like 
their needs are being met as they see the specific mathematics they need to know to teach. 
However, because they are not being exposed to broader mathematics concepts nor to 
their relationships to student thinking, teachers are not necessarily well-prepared to help 
students explore the mathematics more broadly than what is presented in the curriculum. 
(Evaluator, K-8 mathematics LSC) 
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Several other factors may have contributed to the lower ratings for content-focused professional 
development sessions. For example, professional development providers — especially teacher 
leaders — sometimes lacked preparation for adequately addressing content. Further, sessions 
designed to deepen content needs — for example, study groups — require highly skilled 
facilitators, and many LSCs lacked these in adequate supplies. In fact, as noted earlier, some 
evaluators reported that teachers were more apt to use small group sessions designed to deepen 
content knowledge as opportunities to talk about topics other than content — logistics, planning, 
and pedagogy — thus reducing both the time spent on content, and the adequacy of these efforts 
to address content needs. 

Quality of Sessions on Pedagogy and Use of Materials 

A primary goal of LSC projects is to improve mathematics and science instruction through the 
use of high-quality materials in the classroom. In pursuit of this goal, LSCs have acted on the 
premise that content and pedagogy are most accessible to teachers when learned in the context of 
student instructional materials, and have spent substantial time and effort on the materials in 
professional development sessions. As one evaluator noted about an observed session, content 
knowledge was addressed: 

. . .Not as a discrete topic, but in the context of the teachers ’ own exploration and 
investigations of the... curriculum. [This] had two significant implications: the 
mathematics content was worthwhile, and it was presented in a “need to know” context 
that was directly motivated by teachers ’ need to apply their content knowledge 
immediately to their classroom instruction. (Evaluator, 6-12 mathematics LSC ) 

While LSC professional development sessions focused on a number of different purposes, 
sessions intended to prepare teachers to use the designated instructional materials appeared to be 
most common. The higher frequency of sessions on materials use is not surprising, given that 
the implementation of the designated materials is a major goal of LSC projects. Evaluators often 
reported that teachers praised LSC professional development sessions for providing opportunities 
to practice using the materials, noting the relevance, practicality, and usefulness of these types of 
sessions. 

While most common, sessions focusing on learning to use specific materials were less likely to 
be rated as highly as sessions focusing on other purpose (see Table 15); typically, sessions on 
materials were less apt to balance attention to content and pedagogy or provide time for teachers 
to reflect on the activities. Professional development sessions intended to promote reflective 
practice, though observed less often, were more likely to be rated highly than sessions focusing 
on the instructional materials, mathematics/science content, or pedagogical strategies. 

Evaluators’ narrative reports noted some weaknesses in sessions focused on the instructional 
materials that may help explain the lower ratings. Lor example, kit- or module-focused sessions 
sometimes focused solely on the “mechanics” of the materials, and neglected appropriate content 
and pedagogy. These types of sessions also ran the risk of communicating to teachers that 
module- or kit-based instruction was “an end rather than a means” to developing student 
understanding. Evaluators suggested that sessions focused on exploring materials sometimes 
lacked opportunities for teachers to make connections between the activities and the “big 



Horizon Research, Inc. 



38 



December 2006 




picture” — that is, how the specific activities fit into a conceptual framework. Finally, sessions 
on materials use sometimes “ran out of time” for “wrap-up” discussions, which might have 
promoted deeper reflection on specific pedagogical strategies or helped teachers “make sense” of 
how the materials support broader curriculum goals. 

Where providers modeled and explicitly discussed investigative strategies, they were able to 
create an experience in which teachers could both learn through an inquiry approach and 
consider the pedagogical implications for their students, as described in these examples: 

In the Floating and Sinking [session], teachers were working to understand the materials 
as users of the materials. The inquiry was a scientific inquiry as learners of science 
directed or guided at times by the scientist or kit specialists. Upon reflection — either as 
an individual, with a peer, or in whole group — the teachers made scientific meaning, 
scientific understanding, and scientific knowledge from that experience. They then 
shifted their inquiry thinking to that as teachers of science and considered how to employ 
their internalization of scientific inquiry in the instruction and assessment of the child 
learner. (Evaluator, K-8 science LSC) 

This session demonstrated — at an exemplary level — “the whole package. ” The 
facilitators understood extremely well the theoretical and conceptual underpinnings of 
the pedagogy they were trying to model and teach. They were reflective, and explicitly 
so, about all aspects of the pedagogical approaches they were teaching. They made 
connections throughout and the result was a very effective weaving together of theory 
and practice. (Evaluator, K-8 science LSC) 

Explicit attention was paid to the importance of debriefing. “ Debriefing is important for 
problem-solving as a way of sharing divergent learning, ” one facilitator commented, as 
she talked about debriefing in the classroom. Ways to debrief were also modeled. Many 
questions were asked of the teachers, such as, “What did you learn form this problem?” 

“ Did you learn more from doing it yourself or by watching other people solve the 
problem?” (Evaluator, 6-12 mathematics LSC) 

Evaluators also reported some “unevenness” in the modeling and discussion of pedagogy in LSC 
professional development sessions. For example, while LSC workshops were typically designed 
to simulate the classroom, evaluators noted that facilitators sometimes used pedagogy 
appropriate for students without making explicit the intent or goal of the strategy. In these cases, 
professional development providers sometimes “missed their chance” to clarify instructional 
approaches in deliberate and unambiguous ways. Said these evaluators: 

Most of the professional development sessions observed exhibited many aspects of high 
quality professional development that is well aligned with the vision of the 
LSC — investigations, collaborative work, time for reflection, exemplary modeling by 
facilitators, relevant mathematics content focused on conceptual understanding. 

However, a few sessions fell short in at least one aspect. Usually effective modeling was 
missing. In a few instances, collaborative work was missing because the facilitator kept 
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sharing his/her stories about the success and failures with a particular investigation. 
(Evaluator, 6-12 mathematics LSC) 



At times the professional development experience is not as rich as it might be. A cohesive 
whole is still lacking. Teachers need more help understanding the bigger picture — both 
in terms of content and pedagogy — and the implications for their practice. In order to do 
that, they need more opportunities to reflect about what they are doing and why. 
Facilitators are currently helping teachers learn some important math content as well as 
the “vocabulary” of CGI, both of which are important in their own right. But to move to 
the next stage, especially in a context in which teachers are being asked to draw from 
many resources to make their instructional decisions, participants need more help 
expanding and connecting their relatively isolated knowledge of effective pedagogy and 
content to effective implementation in their classrooms. Facilitators need to model and 
be very explicit about how this can happen — both in workshop settings and in working 
with teachers in their own classrooms. (Evaluator, K-8 mathematics LSC) 

Time constraints also caused providers to “slip into show and tell,” at times sacrificing 
understanding in the interest of covering material. In addition, facilitators sometimes paid 
insufficient attention to clarifying how teachers might modify their own teaching, based on what 
they had observed or experienced in the workshop. Thus, while teachers were highly engaged in 
the sessions, the overall quality of the session might have been strengthened if professional 
development providers had allowed teachers to reflect on their experience as “students” and the 
ways in which they might tailor their own lessons accordingly. 

Evaluators also noted the need for “more and better use” of student work to help teachers 
identify misconceptions. Projects that did this effectively provided opportunities for teachers to 
learn new teaching strategies “outside” the modules, but with clear relevance to the pedagogy 
needed for implementing the materials. Finally, evaluators noted the need for LSCs to give more 
attention to assessment, the ways in which students process information, and strategies for 
working with special needs students — all areas of observed weakness among classroom teachers. 
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Part Four 



Impact of LSC on Teachers and Students 



NSF goals for the LSC Initiative include enhancing teachers’ content and pedagogical 
knowledge, and their capacity to use instructional materials — all in the service of improving 
student achievement in mathematics and science. The previous sections have looked at the 
nature and quality of LSC professional development. In this section, we take a look at how LSC 
professional development has influenced the quality of teaching and learning in the classroom. 
Topics include the LSC’s impact on: (1) teachers’ attitudes and beliefs; (2) teachers’ perceptions 
of their content preparedness and their preparedness to teach mathematics/science; (3) the quality 
of classroom instruction as well as teachers’ capacity to use instructional materials and inquiry- 
based practices, and create an investigative classroom culture; and (4) student achievement in 
mathematics and science. Analyses of data from teacher interviews and questionnaires, principal 
questionnaires, classroom observations, and LSC studies on student impact provide the basis for 
findings in this section. 



Impact on Teacher Attitudes and Beliefs 

Data collected as part of the core evaluation indicate that the LSC appears to be having a small, 
positive impact on teacher attitudes. A longitudinal analysis of teacher questionnaire data 7 
(Heck, Rosenberg, & Crawford, 2006a) found that teacher attitudes toward reform-oriented 
teaching were relatively positive across the LSC program, even among teachers who had not yet 
participated in LSC professional development. The researchers also found a small, but 
significant, positive relationship between teacher participation in LSC professional development 
and a composite measuring these attitudes. (See Figure 4.) As a measure of effect size, the 
composite scores of teachers with the mean number of LSC professional development hours (36 
hours) can be compared to the composite scores of untreated teachers. For the attitudes toward 
reform-oriented teaching, the effect size was extremely small, 0.04 standard deviations or a 0.37 
point increase on this composite, an indication that the statistically significant finding is likely a 
reflection of the extremely large sample size. 



7 The longitudinal teacher questionnaire study treated the extent of teacher participation in LSC professional 
development as a continuous variable, ranging from 0 to 200 hours. 
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Predicted Composite Scores: 
Attitudes Toward Reform-Oriented Teaching 



90 




60 80 100 120 140 

Hours of LSC Professional Development 



Figure 4 



Additional analyses of the teacher questionnaire data (Rosenberg, Heck, & Banilower, 2005) 
examined if the effectiveness of LSC professional development varied by whether teachers had a 
strong content background (determined by the number of college level mathematics/science 
courses taken). Although teachers with strong content backgrounds had slightly more positive 
attitudes toward Standards - based instruction (a very small effect size of 0.04 standard 
deviations), the relationship between professional development and this outcome was not 
different for the two groups of teachers. In other words, LSC professional development was just 
as effective for teachers with less college mathematics/science preparation as it was for those 
with more preparation. 

An analysis of cross-sectional LSC teacher interview data also shows a small impact of the LSC 
on teachers’ attitudes. Shimkus and Banilower (2004b) found that about 1 in 5 teachers reported 
that the LSC influenced their vision of effective mathematics/science education, though the 

Q 

likelihood of a teacher reporting this impact was not a function of treatment level. Several 
factors may help explain the fairly small levels of impact found in the analysis of the teacher 
interview data. First, many teachers may have already had a vision of instruction that was 
aligned with the LSC vision, leaving little room for growth; the relatively high scores on the 
“attitudes toward reform-oriented teaching” composite found by Heck et al. (2006a), even among 
untreated teachers, support this hypothesis. In addition, the lack of significance in the teacher 
interview study may be partly due to a number of methodological factors, including the relatively 



8 In the teacher interview study, extent of participation in LSC professional development was categorized into three 
groups: 20-59 hours, 60-99 hours, and 100 or more hours. This grouping was necessitated by the relatively small 
sample size and the relatively gross measure of treatment used on the interview summary form. 
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small sample size 9 , the dichotomous outcome variable (impact/no impact), and the cross- 
sectional (rather than longitudinal) nature of the data. 



Data from evaluator reports give a more positive view of the LSC’s impact on teacher attitudes 
and beliefs. For example, workshop evaluations and interviews indicate that teachers leave 
professional development activities with a greater sense of enthusiasm for teaching, a heightened 
awareness of how students learn, and a willingness to modify their teaching practices and 
collaborate with colleagues. Teachers attributed new beliefs about how students leam — through 
questioning, discovery, and reflection — to their exposure to inquiry-based instruction in LSC 
professional development. According to one evaluator, teachers in the project “expressed a shift 
from ‘static’ and teacher-centered practices to creative and student-centered practices.” Another 
evaluator credited the LSC with altering teachers’ attitudes to the point where “a culture of 
resistance to the new curriculum” had now been replaced by a culture of “acceptance and 
sometimes demand.” Said this evaluator about the changes in attitudes reported by teachers over 
time: 



Teachers reported that the LSC had a positive impact on changing many of their attitudes 
about mathematics instruction. Their attitudes shifted gradually over the three years of 
the LSC toward vcduing and adopting the practices that promote inquiry investigation 
and exploration. Both in classroom obserx’ations and interviews, teachers reported 
changes in their thinking about what constitutes effective mathematics instruction. The 
sample of teachers commented on changes in attitudes toward the accessibility of reform 
mathematics materials and instruction to low performing students, as well as their 
ovemll reception of mathematics instruction. (Evaluator, K-12 mathematics LSC) 

Other evaluators cited changes in teachers’ beliefs about who can learn science and mathematics. 
For example, prevailing attitudes among some teachers before LSC workshops included low 
expectations and the need for ability grouping. LSC professional development helped change 
these beliefs. Said these teachers about the impact of LSC professional development: 

Before IMP, I felt that there were mathematically unreachable students. I felt that 
students could not go on to more challenging ideas like cdgebra, statistics, probability, or 
trig without basic skills. Fortunately, with my IMP training, I have a different feeling 
about students. 1 strongly believe in access to mathematics for all. (Teacher, 6-12 
mathematics LSC) 

I look at student learning completely differently. It opened my eyes to how many different 
ways there are to come to the same answer... The professioncd development experience 
made me realize that kids need time, developmentally, to understand the concepts. 
(Teacher, K-12 mathematics LSC) 

These testimonials suggest that the LSC influenced teachers’ attitudes toward teaching and 
learning in important ways. On the other hand, teacher beliefs were also shaped by state and 



9 The study was based on annual interviews with a sample of 10 teachers per project, as required by the core 
evaluation, compared to the 300 or so who completed questionnaires. 
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local context, and the extent of support for LSC reforms. For example, one evaluator cited few 
changes in teacher beliefs, and actual “drops” in attitudes about the importance of engaging 
students in inquiry-oriented activities. In this case, the evaluator suggested that backsliding 
might be linked to the high stakes testing currently mandated in the state, as well as the “rigorous 
expectations about the breadth of coverage” required at each grade level by the state standards. 
These kinds of factors may have diminished the overall impact of LSCs on teacher attitudes and 
beliefs. 



Impact on Teachers’ Perceptions of Preparedness 

Pedagogical Preparedness 

The LSC appears to be having a positive impact on teachers’ perceptions of their pedagogical 
preparedness. For example, an analysis of longitudinal teacher questionnaire data (Heck, 
Rosenberg, & Crawford, 2006a) found that scores on a composite that measured teachers’ 
perceptions of their pedagogical preparedness increased with teacher participation in LSC 
professional development. (See Figure 5.) The effect size was 0.22 standard deviations, 
equivalent to a 2.94 point increase on this composite. Interestingly, the study also found that, 
controlling for teacher participation in the LSC, scores on the pedagogical preparedness 
composite increased as projects matured, an indication that the LSC may be having a “systemic” 
effect on teacher preparedness. 



Predicted Composite Scores: 
Pedagogical Preparedness 




Figure 5 



The analyses by Rosenberg et al. (2005) found that prior to participating in LSC professional 
development, teachers with stronger content preparation tended to score about six points higher 
(0.27 standard deviations) on this outcome than teachers with less content preparation. However, 
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the LSC was effective at narrowing this gap, with increasing professional development leading to 
a decreasing difference between the two groups. For teachers with the mean number of hours of 
LSC professional development, the gap was reduced to about three points (0.17 standard 
deviations). 

The teacher interview study (Shimkus & Banilower, 2004b) showed mixed results for the LSC’s 
impact on teachers’ perceptions of their pedagogical preparedness. For example, the data 
indicate an upward (though not statistically significant) trend in the relationship between 
teachers’ reports of impact on their pedagogical preparedness and their increasing participation 
in LSC professional development. No relationship was found between the extent of teacher 
participation in LSC professional development and the likelihood of teachers reporting an impact 
on their preparedness to use the designated instructional materials. However, the same caveats 
mentioned earlier regarding the methodological limitations of the teacher interview 
study — relatively small sample size, the dichotomous outcome variable, and the cross-sectional 
nature of the data — apply to these findings as well. 

At the same time, teachers may over-rate their sense of pedagogical preparedness for other 
reasons. Said one evaluator: 

A teacher who currently teaches science with a textbook and has sufficient materials to 
support the activities in the book may report that she is “very well prepared” to teach 
physical science; she is indistinguishable from the teacher who is about to embark on the 
journey of teaching inquiry-centered materials-based science. (Evaluator, K-8 science 
LSC) 

In addition, teachers’ perceptions of their preparedness may change over time, with teachers 
lowering their ratings as they begin to understand the complexities of truly effective instruction. 
One evaluator makes this point well: 

It is not irrelevant in this context to point out that when teachers in [ this district] were 
asked to rate their teaching some time ago, the teachers with the most training who were 
the most effective classroom teachers rated themselves the lowest. Interpreting the 
question of preparedness is partially a question of understanding and perception. 
(Evaluator, K-8 science LSC) 

Mathematics/Science Content Preparedness 

There is also evidence of the LSC’s impact on teachers’ mathematics/science content 
preparedness. The longitudinal teacher questionnaire study found a positive relationship 
between the extent of teacher participation in LSC professional development and scores on a 
composite variable measuring teachers’ perceptions of their content preparedness. (See Ligure 
6.) The difference between teachers with the mean number of professional development hours 
and untreated teachers was very small, 0.15 standard deviations, equivalent to a 2.76 point 
increase on this composite. As was the case with pedagogical preparedness, Heck, Rosenberg, & 
Crawford (2006a) found that after controlling for teacher participation in LSC professional 
development, composite scores for content preparedness increased as projects matured. 
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Predicted Composite Scores: 
Mathematics/Science Content Preparedness 




Figure 6 



Not surprisingly, teachers with more college-level coursework in mathematics/science scored 
significantly higher on this composite, an effect size of 0.40 standard deviations prior to 
participating in LSC professional development (Rosenberg et al., 2005). However, LSC 
professional development was positively related to scores on this composite, regardless of 
teachers’ content background. Furthermore, LSC professional development is associated with a 
narrowing of the gap between the two groups on this outcome (a reduction of 0.02 standard 
deviations for teachers with the mean number of hours of professional development). 

The teacher interview study (Shimkus & Banilower, 2004b) provides further evidence of impact 
as teachers’ professional development hours accrue: although only about 20 percent of treated 
teachers reported an impact on their content knowledge, teachers who had participated in 60 or 
more hours of LSC professional development were more likely to report an impact on their 
content preparedness than teachers with fewer than 60 hours of treatment. 10 

Evaluator reports also cited teacher interview data in support of the LSC’s impact on perceptions 
of content preparedness. For example, participants noted that LSC professional development had 
helped them to develop a better understanding of mathematics/science content and concepts, and 
to “discover” new ways to connect content and “real world” applications. 



10 As noted earlier, the treatment levels utilized for the interview study were: 20-59 hours, 60-99 hours, and 100 or 
more hours. 



Horizon Research, Inc. 



46 



December 2006 




Impact on Classroom Practices 



Content Taught in LSC Districts 

Evidence of the LSC’s impact on the content taught in mathematics/science classes emerged in 
the analysis of classroom observation data (Bowes & Banilower, 2004). The findings indicate 
that teacher participation in LSC professional development is positively correlated with evaluator 
ratings of the quality of mathematics/science content in observed lessons. Specifically, lessons 
of teachers with greater participation in LSC professional development were more likely to be 
rated highly for containing significant and developmentally appropriate content, and for treating 
mathematics/science as a dynamic body of knowledge. 11 (See Table 16.) Similarly, lessons 
based on the designated instructional materials were more likely to be rated highly on these 
indicators. The combination of professional development and use of the designated instructional 
materials appears to have a greater effect than either factor alone. 



Table 16 



Predicted Probability of a Lesson Receiving a Rating in High Category, 
by Hours of LSC Professional Development and Use of Designated Materials 





Predicted Probability 




0 


1-19 


20-39 


40-79 


80 or More 




Hours 


Hours 


Hours 


Hours 


Hours 


Significant and worthwhile content* 












Did not use designated materials 


0.48 


0.55 


0.62 


0.63 


0.58 


Used designated materials 


0.73 


0.78 


0.82 


0.83 


0.80 


Developmentally appropriate content* 

Did not use designated materials 


0.50 


0.53 


0.63 


0.59 


0.57 


Used designated materials 


0.70 


0.73 


0.81 


0.78 


0.76 


Mathematics/science portrayed as a dynamic body of 
knowledge* 

Did not use designated materials 


0.17 


0.19 


0.30 


0.25 


0.28 


Used designated materials 


0.35 


0.37 


0.53 


0.46 


0.50 



* Ratings on this indicator significantly more likely to be high for greater levels of teacher participation in LSC professional 
development and if the lesson was based on the designated materials, p < 0.05. 



Shimkus and Banilower (2004b) found a lower level of impact in the teacher interview study, 
with about 20 percent of treated teachers attributing changes in the content they teach to the 
LSC. The likelihood of a teacher reporting this impact did not change with increasing levels of 
professional development. Given that Bowes and Banilower (2004) found that a majority of 
lessons of untreated teachers were rated highly for the significance and developmental 
appropriateness of the content, this result is not surprising. 



1 1 The classroom observation study used teacher professional development participation data from the teacher 
questionnaire. Although the sample was too small to treat these data as continuous, a greater number of categories 
could be utilized than in the interview study. Thus, the observation data were analyzed by five categories: 0 hours, 
1-19 hours, 20-39 hours, 40-79 hours, and 80 or more hours. 
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As found in the analysis of classroom observation data by Bowes and Banilower (2004), 
evaluators cited evidence of teachers’ growth in content knowledge over time and with 
participation in LSC professional development. Said these evaluators: 

A deeper understanding of the content is reflected in the teacher-student interactions 
observed in classrooms. Teachers are better able to ask questions that have the potential 
for leading to greater conceptual understanding among the students. This would suggest 
greater understanding of the mathematics content. . .Teachers frequently say that their 
understanding of mathematics has been greatly enhanced as a result of learning to use 
the Investigations materials. (Evaluator, K-8 mathematics LSC) 

In spite of the project ’s effort, LSC teachers remain weak in science content knowledge 
...However, we do not find the overall low level of science content knowledge surprising. 
Considering the deep content deficiencies most elementary teachers have, it is not 
reasonable to expect the current effort to produce immediate, large-scale results. 
Moreover, there is some indication from the classroom obserx’ation data that progress is 
being made. The two obserx’ations in which the content dimension was rated highest (4) 
were conducted in the classroom of teachers who had the most extensive professional 
development experiences with the LSC. We conclude that although current 
understanding of content is low, as teachers participate in the content offerings [the LSC] 
provides the benefits to their knowledge will begin to accrue. (Evaluator, K-8 science 
LSC) 

Still, evaluators’ narrative descriptions indicate that teachers’ capacity to address content ranged 
widely in observed LSC classes. In some cases, evaluators saw no evidence of “content 
expansion,” and pointed to classroom discussions that lacked rigor, as teachers encountered 
discomfort moving into content areas that were still new for them. Said these evaluators: 

[TJeachers attending both kinds of [content] sessions overwhelmingly indicated in both 
interviews and surveys that they felt their own understanding of mathematics content had 
been enhanced, and that they were much more confident about teaching mathematics . . . 
The 16 classroom observations conducted this year, however, did not provide clear 
evidence that teachers have deepened their content knowledge. (Evaluator, K-8 
mathematics LSC) 

The discrepancy between [survey] data and data collected during observations of 
classrooms this year can be explained simply. The [LSC] is succeeding in deepening 
participants ’ content knowledge, but much work remains to be done before teachers are 
able to guide their students through an in-depth exploration of the science concepts 
embedded in the science kits. (Evaluator, K-8 science LSC) 

One evaluator noted that, “content was consistently rated lower than pedagogy and instructional 
materials,” and while content was accurate and developmentally appropriate, it rarely engaged 
students in “key ideas.” Other evaluators cited evidence of “continuing misconceptions” in 
content among teachers, and gave examples of teachers’ difficulty in “translating” their content 
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knowledge into student learning. As with pedagogy, evaluators reported wide discrepancies in 
teachers’ actual and perceived content preparedness. 

Teachers’ Use of Reform-Oriented Instructional Practices 

Evidence from the teacher interview study points to a link between participation in LSC 
professional development and changes in teachers’ use of reform-based pedagogy. Shimkus and 
Banilower (2004b) reported, for example, that half of the teachers that participated in LSC 
professional development indicated that the LSC had influenced their selection of instructional 
strategies. Lurther, teachers with 60 or more hours of treatment were more likely to report this 
impact than teachers with fewer than 60 hours of treatment. Similarly, although relatively few 
teachers reported that the LSC had an impact on their choice of assessment strategies, teachers 
with 60 or more hours of LSC professional development were more likely to indicate an impact 
in this area. 

Heck, Rosenberg, & Crawford’s (2006a) analyses of teacher questionnaire data, utilizing more 
precisely measured treatment data, also found a relationship between participation in LSC 
professional development and teachers’ reported frequency of use of investigative classroom 
practices. This relationship is non-linear, as shown in Ligure 7, with most of the impact 
occurring at 80 or fewer hours of treatment. Teachers with the mean number of professional 
development hours scored, on average, 7.35 points higher on this composite than untreated 
teachers, an effect size of 0.53 standard deviations. This same study found no relationship 
between participation in LSC professional development and teachers’ reported frequency of use 
of traditional classroom practices. Rosenberg et al. (2005) found that teachers with greater 
college-level content preparation scored slightly higher on the investigative practices composite 
than teachers with less content preparation (an effect size of 0.18 standard deviations). However, 
the overall relationship between LSC professional development and this outcome was the same 
for both groups. 



Predicted Composite Scores: 
Use of Investigative Practices 




Figure 7 



Horizon Research, Inc. 



49 



December 2006 





Evaluators noted that teachers frequently reported that the LSC had increased their confidence in 
teaching mathematics and science — a confidence that observations often revealed to be 
“misplaced.” In part, the discrepancies between teachers’ self-reported data on preparedness to 
use reform-based instructional strategies, and their actual teaching behaviors may stem from 
their limited understanding of the depth of change that LSCs seek, and the “false” sense of 
preparedness teachers sometimes feel about applying new practices. One evaluator explained the 
phenomenon this way: 

This sense of preparedness that teachers have may reflect the success of their 
professional development experiences. We want to suggest, however, that the relatively 
constant and high proportion of teachers reporting this sense of preparedness may 
indicate that it is a false sense of preparedness on the part of teachers. That is, we 
suspect that teachers do not fully understand the difficulty of appropriately implementing 
the designated instructional materials, let alone the difficulty of teaching for 
understanding. Therefore, they are reporting preparedness for something other than 
what they are being asked in reality to do. That the classrooms we observed did not 
generally reflect high levels of appropriate pedagogy and that teachers did not reflect an 
awareness of this in pre- and post-obseri’ation interviews suggests to us that many 
teachers do not have a good understanding of what effective pedagogy entails. 

(Evaluator, 6-12 mathematics LSC) 

Still, evaluators’ descriptions of classroom observations early and late in the life of LSCs 
supported the notion of growth over time in teachers’ use of instructional practices aligned with 
the LSC vision. Lor example, observations early in a project often revealed the “mechanical” 
use of instructional materials and strategies; with time and practice, evaluators cited evidence of 
“continual” improvements in both the use of the designated instructional materials and teachers’ 
ability to let students actively participate in the learning process. In examples of high quality 
instruction, teachers posed ideas and approaches, engaged students through questioning, and 
facilitated the process of investigation. Said one evaluator of a project that began in 1995: 

Based on teacher behavior during classroom observations during 1999, it was clear that 
teachers now knew that asking the right question is often a more valuable instructional 
technique than supplying the right answer. The improvement in questioning skills was 
notable. (Evaluator, K-8 science LSC) 

A program- wide analysis of classroom observation ratings supports individual evaluators’ 
narrative descriptions of their observations. Bowes and Banilower (2004) found that the extent 
of teacher participation in LSC professional development and the use of the designated 
instructional materials are positively correlated with evaluator ratings of the quality of teachers’ 
questioning and sense-making strategies. (See Table 17.) The data also show, however, that 
fewer than half of the lessons, even those of the most highly treated teachers, were likely to be 
rated highly in these two areas. 
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Table 17 



Predicted Probability of a Lesson Receiving a Rating in High Category, 
by Hours of LSC Professional Development and Use of Designated Materials 







Predicted Probability 






0 

Hours 


1-19 

Hours 


20-39 

Hours 


40-79 

Hours 


80 or More 
Hours 


Teacher’s questioning strategies likely to enhance 
student understanding/problem solving* 

Did not use designated materials 


0.21 


0.23 


0.32 


0.28 


0.30 


Used designated materials 


0.35 


0.36 


0.48 


0.44 


0.46 


Appropriate “sense-making”* 

Did not use designated materials 


0.19 


0.18 


0.31 


0.27 


0.24 


Used designated materials 


0.30 


0.27 


0.44 


0.40 


0.36 



* Ratings on this indicator significantly more likely to be high for greater levels of teacher participation in LSC professional 
development and if the lesson was based on the designated materials, p < 0.05. 



Descriptions of classroom observations provided by evaluators reinforce the somewhat mixed 
findings on the LSC’s impact on teachers’ use of reform-oriented instructional practices. For 
example, observations frequently revealed both “rudimentary” and “accomplished” examples of 
mathematics and science instruction among LSC teachers. Many teachers demonstrated the 
ability to implement “pieces of reform,” but few had yet to deliver the “whole package” to their 
students. Said these evaluators: 

The most salient evidence that the [LSC’s] high quality professional development was 
somewhat limited in its capacity to facilitate high quality instruction was the continuing 
dependence on traditional teacher-directed instruction demonstrated by many of the 
teachers obseri’ed in their classrooms. Although these teachers utilized some effective 
questioning strategies, and facilitated class discussions to assess student understanding 
of key mathematical concepts, it was almost always facilitated as a whole group lesson 
and not all students participated. These teachers never left the front of the classroom nor 
did they incorporate opportunities to give students the time or autonomy for independent 
thinking and problem solving which many of the students in these classrooms 
demonstrated they were more than ready to do. (Evaluator, 6-12 mathematics LSC) 

Nearly all observations involved active student participation in some kind of activity and 
some level of class discussion about the mathematics. However, the quality of this 
discussion varied widely. No observed teachers were able to maintain discussion at a 
high conceptual level — this has not been effectively modeled for them in the professional 
development. However, most teachers were eliciting student input at some level above 
what they had been doing before, according to their own reporting. Likewise, 
collaborative learning has been adopted by nearly all teachers at some level, but most 
teachers still struggle to find ways to help their groups work together effectively. 
(Evaluator, K-8 mathematics LSC) 

The lessons rated at the beginning stages of effective instruction were good science 
lessons in terms of design and implementation but they were not engaging or motivating 
for students. Mostly, the concepts or objectives got lost in the activity. . . Students were 
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not asked to think about the science.... These lessons were lacking in content, purpose, 
and engagement. (Evaluator, K-8 science LSC) 

Case studies conducted by 10 LSC Pis (Pasley, 2002) also revealed some weaknesses in LSC 
teachers’ instruction. Based on classroom observations. Pis cited examples of limited student 
engagement with the concepts in the modules; “mechanical” implementation of the instructional 
materials; infrequent use of higher- level questioning strategies or sense-making — i'or example, of 
data students had collected in their investigations; and limited discussion of the meaning behind 
particular activities or how these activities fit into the "big picture" of the unit. Evaluators’ 
reports supported these findings, citing questioning strategies, “sense making,” and “closure” as 
particularly challenging for teachers. Said one evaluator: 

The structure of this lesson incorporated cooperative learning and decision making, 
journal writing, and presenting results — all elements of a good activity. But it lacked 
context and purpose. Students seemed a little confused about what they were doing and 
eventually drifted off task. There was no closure to the lesson. (Evaluator, K-8 science 
LSC) 

Many of the changes in instruction cited by teachers were, in the words of one evaluator, 

“surface changes only.” 

In some classes, teachers were too procedure-focused, consequently reducing the investigative 
nature of the materials. Others equated any kind of hands-on instruction with inquiry-based 
learning, thereby missing key opportunities to make conceptual connections. Some descriptions 
of classroom observations pointed to “superficial” questions focused on vocabulary, “hurried 
summaries of what happened,” and limited teacher success in developing conceptual knowledge. 
Said these evaluators: 

A common difficulty was helping students derive meaning from obserx’ations and data. 
Among those receiving lower ratings, the most prominent weaknesses in the lessons 
related to conceptual clarity, including inadequately drawn connections among key 
concepts, and/or a lack of clear connections relating the results of the investigations to 
the concepts being explored. Such problems tended to arise from the teacher’s 
unfamiliarity with the inquiry process, lack of comfort with certain underlying concepts, 
or some combination of the two. (Evaluator, K-8 science LSC) 

These data, combined with our classroom observation data, indicate that many of the 
teachers in the district are currently at the level of mechanical use in teaching the science 
kits. They are comfortable guiding students through the lessons and do this with 
proficiency — much like the facilitators of the Introductory Kit workshops. They rarely, 
however, are able to use pedagogiccd strategies intended to help students develop 
conceptual understanding. (Evaluator, K-8 science LSC) 

Evaluators also cited some glaring differences in teacher self-reported changes in their 
instruction and their actual classroom practices. For example, in an example cited by an 
evaluator of a K-12 mathematics LSC, teachers explained in interviews prior to observations 
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about the importance of group learning, but subsequently conducted lessons in which they were 
simply presenting information, with no opportunity for students to engage in any cooperative, 
participatory learning. These examples and others suggest that the LSC has been successful in 
shifting teachers’ thinking around pedagogy; however, the program has had a more limited 
impact on teachers’ ability to fully operationalize what they are learning in this area. 

Teachers’ Use of Instructional Materials 

A critical feature of the LSC program is the focus of professional development around 
instructional materials, where teachers engage in the activities as students might, with guided 
discussions facilitated by project staff, teacher leaders or content experts. As described 
previously, a number of studies have found that the use of district-designated instructional 
materials is positively correlated with several key outcomes, including the more frequent use of 
investigative classroom practices, and a greater emphasis on important and developmentally 
appropriate mathematics/science. Analysis of teacher interview data (Shimkus & Banilower, 
2004b) indicates that teachers with 60 or more hours of LSC professional development were 
more likely to report an impact on their use of the designated instructional materials than 
teachers participating in fewer than 60 hours. 

Similarly, longitudinal analysis of the teacher questionnaire data (Heck, Rosenberg, & Crawford, 
2006b) shows that teachers tended to use the designated instructional materials more frequently 
as their participation in LSC professional development increased. 12 (See Figure 8.) For 
example, teachers who had not yet participated in the LSC were predicted to have a 27 percent 
probability of using the designated instructional materials at least once a week, compared to a 70 
percent probability for a teacher with the mean number of professional development hours. As 
was the case with other outcomes, most of the change occurred in the first 80 hours of 
professional development. Additionally, the researchers found that even after controlling for 
teacher participation in LSC professional development, teachers’ reported use of the designated 
instructional materials increased as projects matured — again, an indication of a “systemic” 
impact. 



12 As noted earlier, the larger sample size and the measurement of treatment in the questionnaire study than in the 
interview study allowed for a more precise examination of the relationship between professional development hours 
and the outcomes. 
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Probability of Use of District-Designated Instructional Materials 



i 

0.8 




0.2 




Hours of LSC Professional Development 



□ Never/Rarely □ Sometimes □ Often HAH Lessons 



Figure 8 



A cross-sectional analysis of principal questionnaire data (Crawford & Banilower, 2004) also 
suggests that the LSC is having a positive impact on teachers’ use of instructional materials. In 
both mathematics and science projects, principals indicated that, over time, the targeted teachers 
in their schools were more likely to be participating in LSC professional development activities 
and to be implementing at least some of the designated instructional materials. (See Figure 9.) 

In mathematics projects, the predicted probability of 60 percent or more of a school’s 
mathematics teachers participating in LSC professional development was 38 percent in a 
project’s Baseline Year and 81 percent in the project’s fifth year. Similarly, the probability of 60 
percent or more of a school’s mathematics teachers using at least some of the designated 
instructional materials at these two time points was 42 and 84 percent respectively. In science 
projects, the probabilities of 60 percent or more of a school’s science teachers participating in 
LSC professional development and using the designated instructional materials during the 
Baseline Year were 67 and 72 percent respectively. In a project’s fifth year, these probabilities 
increased to 93 and 94 percent respectively 
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Principal Questionnaire Outcomes, by Project Year 

Teachers Involvement in LSC Professional Development Activities 





Teachers Implementing at Least Some of the District-Designated Instructional Materials 





Figure 9 



As with content and pedagogy, evaluators also commented on teachers’ increasing capacity over 
time to teach the materials as designed. Observations also suggest that, while the use of 
instructional materials was fairly widespread in observed classrooms of LSC teachers, the quality 
of use varied tremendously. By no means did use of the designated instructional materials 
necessarily translate into a well-implemented lesson. Said one evaluator: 
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Perhaps the most significant finding lies in combined classroom observation data. Based 
on the classroom observation data, 15 of 16 observations were conducted with teachers 
using the materials supported by the LSC. Of those 15, 8 teachers conducted lessons that 
focused on procedurcd knowledge rather than conceptual knowledge. (Evaluator, K-8 
mathematics LSC) 

Evaluators from other LSCs drew similar conclusions from their observations: 

[Teachers] are using the kits, but are not yet at the point of selecting and tailoring 
appropriate kit activities to meet their goals or being able to extend beyond pre-planned 
suggestions that come with the kits. In a sense, some teachers are using the kits with a 
“ recipe ” approach... One positive effect of the program is that teachers recognize the 
difference between hands-on activity in the science classroom and authentic, rigorous 
investigation. They realize that they are becoming more capable in the hands-on part of 
science exploration, but they cdso recognize that they are less fluent with the minds-on 
part of exploration. (Evaluator, K-8 science LSC) 

In the early years of the project, teachers would often point out the basic notions of the 
content embedded in the kit to their students, but they would miss opportunities, as they 
arose, to tease out further (mis [understandings, often because they seemed not entirely 
comfortable with the content themselves. This year, more teachers took advantage of 
opportunities to enrich students’ knowledge in the course of a lesson. The majority of 
teachers are still considered at the “mechanical” stage of their kit use, but it is solid 
mechanical use. (Evaluator, K-8 science LSC) 

Over the last two years, as project staff have monitored the level of curriculum 
implementation in elementary and middle schools, they have increasingly focused on the 
distinction between “using the materials ” and “implementing as intended. ” While large 
numbers of teachers are implementing the materials... this use was too often sporadic or 
selective... not teaching the materials as they were intended by the designers... More often 
than not, staff diagnosed the difficulty as unfamiliarity or discomfort with the 
instructional model implicit in the curriculum. (Evaluator, K-12 mathematics LSC) 

According to evaluators, teachers in the early stages of learning were more likely to use the 
materials mechanically, or to modify them in inappropriate ways. Some teachers “jumped the 
gun,” skipping to activities on higher level concepts without an adequate foundation for students. 
Still others omitted “rich” activities, revised lessons, or added supplementary materials that 
“short-cut the development of conceptual understanding as laid out in the curriculum.” These 
adaptations often reflected teachers’ concerns about time constraints and classroom management, 
and resulted in more traditional approaches or materials. 

Teachers’ lack of content knowledge also seemed to limit their use of the materials in 
appropriate ways. As a result, while many teachers engaged students with the materials, the 
lessons had little student-to-student discussion and very limited questioning to promote concept 
development around the activities. Still, while use of the materials may be far from perfect 
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among many LSC-treated teachers, classroom observations suggest that the use of the materials 
has enhanced the quality of instruction. Said this evaluator: “With the selection and instruction 
of the exemplary curricula in Year Three, there was a noticeable improvement in the quality of 
mathematics instruction across the district.” 

Classroom Culture 

In addition to influencing the content taught, teaching strategies, and instructional materials used, 
there is evidence to suggest that the LSC has had an impact on the culture of mathematics and 
science classrooms. For example, the longitudinal analysis of teacher questionnaire data (Heck, 
Rosenberg, & Crawford, 2006a) found that the amount of time spent in LSC professional 
development is associated with higher scores on a composite measuring practices that foster an 
investigative classroom culture. (See Figure 10.) The typical LSC participant scored 6.21 points 
higher on this composite than the typical non-participant, an effect size of 0.48 standard 
deviations. In addition, teachers with more college-level coursework in mathematics/science 
tended to score higher on this outcome (an effect size of 0.1 1 standard deviations), though the 
overall pattern of growth was the same for both groups. As with the other outcomes, most of the 
gain occurred in the first 80 hours of treatment. 



Predicted Composite Scores: 
Investigative Culture 




Hours of LSC Professional Development 



Figure 10 



Evaluator ratings of observed lessons support this finding. In their analysis of classroom 
observation data, Bowes and Banilower (2004) found that lessons were more likely to be rated 
highly for actively involving all students, engaging students intellectually, and creating a climate 
of respect and rigor when taught by teachers with higher levels of LSC professional 
development. Lessons based on the designated instructional materials were more likely to be 
rated highly in these areas as well. (See Table 18.) Again, the combination of professional 
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development and use of the designated instructional materials appears to have a greater affect 
than either factor alone. 



Table 18 



Predicted Probability of a Lesson Receiving a Rating in High Category, 
by Hours of LSC Professional Development and Use of Designated Materials 







Predicted Probability 






0 

Hours 


1-19 

Hours 


20-39 

Hours 


40-79 

Hours 


80 or More 
Hours 


Active participation of all encouraged and valued* 

Did not use designated materials 


0.56 


0.56 


0.67 


0.67 


0.65 


Used designated materials 


0.63 


0.64 


0.73 


0.73 


0.72 


Climate of respect for student’s ideas, questions, and 
contributions* 

Did not use designated materials 


0.57 


0.62 


0.67 


0.64 


0.67 


Used designated materials 


0.62 


0.67 


0.72 


0.69 


0.72 


Students intellectually engaged* 

Did not use designated materials 


0.23 


0.29 


0.35 


0.32 


0.30 


Used designated materials 


0.39 


0.46 


0.53 


0.50 


0.48 


Intellectual rigor, constructive criticism, and the challenging 
of ideas evident* 

Did not use designated materials 


0.19 


0.21 


0.27 


0.24 


0.23 


Used designated materials 


0.31 


0.34 


0.42 


0.38 


0.37 



* Ratings on this indicator significantly more likely to be high for greater levels of teacher participation in LSC professional 
development and if the lesson was based on the designated materials, p < 0.05. 



Evaluators’ descriptions of classroom observations also suggest that the LSC is having an impact 
on teachers’ proficiency in promoting an investigative classroom culture. Evaluators cited 
examples where teachers showed increasing comfort and skill in the investigative process, where 
instruction was more interactive, where students were on task at higher levels of engagement, 
and where students were no longer working in isolation and were offering more conjectures of 
their own. The following examples illustrate the high end of the scale where this kind of 
classroom culture was occurring: 

[In observed lessons,] teachers incorporated problem-solving investigation, questioning 
to elicit students’ thinking and understanding; cooperative learning and group 
investigation; group presentations; small and large group reflection and discussion. The 
teachers acted as facilitators of the lesson, circulating from group to group responding to 
students’ questions; facilitating discussions and making sense of the mathematics to 
small groups of students, and to the class as a whole. The teachers were cognizant of 
students ’ needs and modified the lesson accordingly, slowing down the lesson to cover 
less material than planned in order to meet students ’ pace and need to understand the 
mathematics. (Evaluator, 6-12 mathematics LSC) 

At the highest level of questioning practices observed in both elementary and secondary 
classrooms, there was an almost seamless discourse between teacher and students that 
advanced mathematics instruction to the level of unrestricted intellectual mathematics 
exploration. In a fourth grade classroom, the teacher and students discussed angles and 
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lines for a full 20 minutes. Students, as well as the teacher, posed questions, offered 
proofs, considered hypotheses. It was the students ’ questions that shaped the discussion, 
placing the teacher in the role of the facilitator.... In this classroom, as in two second 
grade classrooms observed, this kind of math discourse was clearly routine.... The 
learning environment created in these classrooms was one of calm, deliberate, 
unrestricted exploration. (Evaluator, K-12 mathematics LSC) 

Students worked in small groups as they designed their investigations. The teacher 
roamed from group to group interacting with the students, asking them questions. She 
was effective in modeling collaborative behavior. When she addressed the whole class, 
she said “Zach and I were talking a minute ago. We were trying to determine how to get 
the water to go through here. ” ...As the children experimented, the teacher helped them 
see the connection between the fabric’s water resistance and its effectiveness in the min. 
(Evaluator, K-8 science LSC) 

On the other hand, there were also many examples where instruction was far from investigative 
in nature. For example, in one class, a teacher set the stage for an inquiry-based lesson with 
appropriate questioning techniques to encourage students to discuss what they already knew 
about the concept of melting. When students showed a lack of understanding about the concept, 
the teacher abandoned the use of questions and resorted to telling students what melting was 
before they began their investigations. She then gave out directions on how students could 
design and conduct their activity to melt ice, rather than giving students the opportunity to 
generate their own methods. Thus, while the teacher initiated the activity with questioning to 
engage students, she relapsed into “familiar” teaching behaviors that precluded any kind of 
investigative culture in the classroom. Evaluators suggest that lack of content knowledge, lack 
of confidence and experience in using the investigative model and/or the materials, and 
resistance to forsaking teacher-directed approaches were among the factors that limited teachers’ 
capacity to create an investigative classroom culture. 



Impact on Quality of Instruction 

Analyses of classroom observation data indicate that the LSC program has had a positive impact 
on the overall quality of mathematics/science instruction. Bowes and Banilower (2004) show 
that the extent of teacher participation in LSC professional development and the use of the 
designated instructional materials are positively associated with higher evaluator ratings of 
lesson quality. Lessons that were taught by teachers with at least 80 hours of treatment and 
based on the designated materials were more than twice as likely to receive a high rating than 
lessons of untreated teachers who were not using the district-designated instructional materials. 
(See Table 19.) 
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Table 19 



Predicted Probability of a Lesson Receiving a Rating in High Category, 
by Hours of LSC Professional Development and Use of Designated Materials 





Predicted Probability 


0 

Hours 


1-19 

Hours 


20-39 

Hours 


40-79 

Hours 


80 or More 
Hours 


Capsule Rating* 

Did not use designated materials 
Used designated materials 


0.15 

0.28 


0.18 

0.33 


0.27 

0.46 


0.21 

0.38 


0.22 

0.39 | 



* Ratings on this indicator significantly more likely to be high for greater levels of teacher participation in LSC professional 
development and if the lesson was based on the designated materials, p < 0.05. 



Bowes and Banilower (2004) also found that lessons in which teachers implemented their 
instructional materials as written, regardless of whether they were the district-designated 
instructional materials, tended to be rated higher than lesson in which teachers heavily modified 
the materials. In fact, the greater the level of adherence to the written materials, the greater the 
probability that the lesson would be rated highly. This pattern was found at all levels of teacher 
participation in LSC professional development. (See Figure 11.) 



Overall Rating of Lesson Quality 



1.00 -| 
0.80 - 
0.60 - 
0.40 - 
0.20 - 
0.00 -■ 



0.62 




0 Hours of LSC PD 80 or More Hours of LSC PD 



□ Materials Not Used □ Materials Used - Low Adherance 

□ Materials Used - Medium Adherance ■ Materials Used - High Adherance 



Figure 11 



These results were confirmed in a structural equation model of LSC teacher questionnaire and 
classroom observation data that took into account the mediating effects of teacher attitudes, 
preparedness, and perceptions of principal support (Banilower & Germuth, 2004). 13 The 



13 The structural equation model study utilized professional development participation data from the teacher 
questionnaire, and treated these data as continuous. 



Horizon Research, Inc. 



60 



December 2006 



analyses examined these relationships by project subject/grade-range and found the same 
patterns in the different targeted areas. (It is important to note that the lack of statistical 
significance for the “use of district-designated instructional materials in observed lesson” 
coefficient in grades 6-12 mathematics projects is likely due to the relatively small sample size 
available for analysis.) In addition, the analysis found that regardless of whether the specific 
observed lesson was based on the designated materials, lessons of teachers that reported using 
these materials more frequently on the teacher questionnaire were more likely to be rated highly. 
(See Table 20.) 



Table 20 



Structural Equation Model Regression Coefficients Predicting 
Observed Lesson Quality, by Project Subject/Grade Range 1 





Regression Coefficient 


Number of hours of LSC professional development 




K-8 Science 


0.11* 


K-8 Mathematics 


0.11* 


6-12 Mathematics 


0.13* 


Use of district-designated instructional materials in observed lesson 




K-8 Science 


0.29* 


K-8 Mathematics 


0.29* 


6-12 Mathematics 


0.20 


Frequency of use of district-designated instructional materials 




K-8 Science 


0.18* 


K-8 Mathematics 


0.18* 


6-12 Mathematics 


0.22* 



* p < 0.05 

f Analysis for grade 6-12 science was omitted due to the extremely small number of cases. 



Analysis of data from classroom observations (Bowes & Banilower, 2004) also indicate that 
lessons taught by teachers who had participated in at least 20 hours of LSC professional 
development were more likely to be judged by observers to be strong in a number of areas, 
including the extent to which: 

• The mathematics/science content was significant and worthwhile; 

• Teacher-presented information was accurate; 

• There was a climate of respect for students' ideas, questions, and contributions; 

• Students were intellectually engaged with important ideas relevant to the focus of the 
lesson; 

• Intellectual rigor, constructive criticism, and the challenging of ideas were valued; 

• The degree of closure or resolution of conceptual understanding was appropriate for 
the developmental levels/needs of the students and the purposes of the lesson; and 

• The teacher's questioning strategies were likely to enhance the development of student 
conceptual understanding — for example, teachers emphasized higher order questions, 
appropriately used “wait time,” identified prior [misjconceptions. 
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The LSC also appears to be having an impact on the amount of time spent on science teaching in 
self-contained classes in the elementary grades. For example, Heck, Rosenberg, & Crawford 
(2006b) found a positive relationship between the extent of teachers’ participation in LSC 
professional development and the number of minutes per week teachers spent on science 
instruction. (See Figure 12.) For a teacher who has received the mean amount of LSC 
professional development, there is a medium-sized effect of 0.45 standard deviations or 26.82 
minutes per week. The researchers also found that teacher participation in LSC professional 
development fully explained the apparent increase in instructional time devoted to science as 
projects matured — that is, there was no “systemic” effect on instructional time for teachers in 
LSC districts who had not participated in LSC professional development. Rosenberg et al. 

(2005) found that this pattern existed for teachers regardless of college content preparation. Still, 
elementary teachers with a greater level of science preparation in college tended to devote more 
time to science teaching (an effect size of 0.17 standard deviations). 



Time Devoted to Science Instruction 
in Elementary Science Classes 




Figure 12 



Impact on Students 

Data on the LSC’s impact on students is quite limited at this time. While some projects included 
a design for assessing impact on students, NSF did not require LSCs funded in the first four 
years of the program to examine impact in this area. Of the 58 projects funded during this time, 
16 provided the results from student impact studies. NSF required projects funded in Cohort 
Five or later to study their impact on students; however, none of these projects have completed 
their studies to date. Reviews of the 16 studies conducted by LSC projects, data from a cross- 
site study of the impact of the LSC on student achievement in elementary science, LSC teacher 
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interview data, and principal questionnaire data have provided some information on the LSC’s 
impact on students. 

Three reviews of the studies provided by LSC projects suggest that LSCs are having a positive 
impact on students. For example, 12 of the 13 studies reviewed by Banilower (2000) and 10 of 
the 12 reviewed by Zhang and Wang (2002 ) 14 , and 5 of the 6 reviewed by Zhang and Frechtling 
(2005) provide evidence of impact on student achievement, attitudes, and/or course taking 
patterns in both mathematics and science LSCs. However, all three reviews noted that due to 
various constraints, the research designs utilized in many of these studies made it difficult to 
judge whether the positive findings are due to the LSC, due to factors other than the LSC, or 
whether they are simply artifacts of the study’s methodology. For example, many of the studies 
did not include a comparison group; studies that did include a comparison group tended to use a 
post-test only design — with no assurances that the groups were equivalent prior to the 
intervention — or did not address potential selection biases. 

Data from the first two years of a three-year cross-site study examining the LSC’s effect on 
student achievement in elementary science has provided some additional evidence of student 
impact. The study was designed as one of several efforts to improve the quantity and quality of 
student impact data; it used a quasi-experimental pre-test/post-test design with the goal of 
determining how teacher participation in LSC professional development affected student 
achievement. Pis of elementary science LSCs helped develop an assessment, which was piloted 
prior to the beginning of the study. Projects could elect to have all classes in a selected grade 
level (4 th , 5 th , or 6 th grade) participate in the study, or they could have a random sample of classes 
participate, with a minimum of six classes per project. The pre-test was administered near the 
beginning of the school year, and the post-test near the end of the school year. Teachers 
completed pre- and post-test questionnaires that provided data on participation in LSC 
professional development, amount of instructional time devoted to science, and the extent to 
which science instruction was based upon the designated instructional materials. 

Results from the first year of the cross-site study (Banilower, 2002) indicated that the extent of 
teacher participation in LSC professional development was a significant, positive predictor of 
student achievement, controlling for prior achievement and a number of student and classroom 
demographics. However, due to problems in data collection, the analysis did not include the 
amount of science instruction students received, or the extent to which that instruction was based 
upon the designated instructional materials. 

The second year of the cross-site study (Germuth & Banilower, 2004) found that teacher 
participation in LSC professional development was a positive predictor of the amount of 
instructional time devoted to science and the extent of use of the designated instructional 
materials. However, neither of these factors was a significant predictor of student achievement. 
In both of these studies, projects were typically unable to require teachers to participate. As a 
result, participating teachers were essentially volunteers, and the possibility of selection bias in 
the study was a concern. In addition, only 12 of the 47 projects that targeted elementary science 



14 Five of the studies reviewed by Zhang and Wang were included in the Banilower review. 
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participated in the first year of the study, and 10 in the second year, increasing the potential for 
selection bias and making it difficult to generalize the results to the LSC program as a whole. 



The teacher interview study (Shimkus & Banilower, 2004b) found that teachers that participated 
in 100 or more hours of LSC professional development were more likely to report that the LSC 
was having a positive impact on students than teachers with fewer hours of treatment (a 17 
percent probability of reporting this impact as compared to an 1 1 percent probability.) Finally, a 
cross-sectional analysis of principal questionnaire data indicates that school administrators in 
LSC districts perceived their schools to be making progress in moving toward excellence in 
mathematics/science education and that student achievement in the targeted subject(s) was 
improving. (See Figure 13.) Results were similar for both mathematics and science projects. 
The predicted probability of a principal indicating the school was at least well along in 
improving it’s mathematics/science program during the Baseline Year was 51 percent for 
mathematics and 59 percent for science, compared to 70 and 82 percent respectively during year 
five. There was a 63 percent probability in mathematics and 78 percent probability in science of 
a principal reporting that student achievement was at least somewhat improved during the 
Baseline Year, compared to 88 and 92 percent respectively during Year Five. 
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Principal Questionnaire Outcomes, by Project Year 

School’s Progress in Moving Toward Excellence in Mathematics/Science Education 





Achievement of Students in this Sc hool Compared to Five Years Ago 



Mathematics 




Science 




2 3 4 

Project Year 



□ Much/Somewhat Worse □ About the Same 

□ Somewhat Improved ■Much Improved 



Figure 13 



Taken as a whole, the available data suggest that the LSC program has had a positive impact on 
students. Additional data from the studies of projects funded after year four (when they are 
completed), as well as data from the third year of the cross-site elementary science program 
study, should shed additional light on the extent of the LSC’s impact on students. 
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Part Five 



Impact of LSC on District Systems 



The primary goal of the LSC Initiative is to improve classroom instruction through teacher 
enhancement. However, NSF also expects projects to develop a supportive context for reform to 
increase the likelihood that LSCs can scale up the interventions during the grant and sustain them 
beyond the life of the grant. Building support among an array of stakeholders, aligning policy 
and practice, developing organizational structures and partnerships, developing capacity, and 
leveraging resources all contribute to efforts to influence the system as a whole in support of the 
LSC vision. This section looks at the extent of support LSCs built for reform, and the likelihood 
that interventions initiated by the LSCs will be sustained over time. 



Development of a Supportive Context for Reform 

The core evaluation asked Pis and evaluators to collaborate on a number of ratings to assess the 
LSCs’ success in building a supportive context for reform. At the end of a project’s Baseline 
Year (project year zero), Year Two (project year two), and Final Year (typically, project year 
five), LSC evaluators and Pis rated the extent to which a variety of district policies — for 
example, curriculum frameworks, selection of instructional materials, and district-wide 
assessments — facilitated or inhibited the process of reform in LSC districts. Pis and evaluators 
also rated the extent to which various stakeholder groups such as parents, teachers, and principals 
supported the LSC reforms. A composite measuring each of the two constructs was created 
based on responses to each set of items at each of these time points. Shimkus and Banilower 
(2004a) examined how project ratings evolved over time. As shown in Figure 14, the district 
policy composite indicates that district policies tended to facilitate LSC reforms, even at the 
beginning of LSC projects, and became more supportive as projects matured (an effect size of 
0.25 standard deviations). The stakeholder support composite shows a similar trend: support for 
LSC reforms tended to be positive initially and to increase over time (an effect size of 0.19 
standard deviations). 
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Extent of Support for the LSC Reforms Composites 




— — Alignment of District Policies* Stakeholder Support* 

* Composite score changes significantly over time, p < 0.05. 

Figure 14 



In addition to the individual indicator ratings included in the composites, evaluators and Pis 
provided an overall rating of the extent to which there was a supportive environment for the LSC 
reforms, using a scale ranging from “Level 1: Predominance of non- supportive context” to 
“Level 5: Exemplary supportive context.” The analysis of these continuum ratings show that 
most LSC projects began their work in contexts already transitioning toward the LSC vision for 
effective mathematics/science education (an implicit criterion for receiving an LSC award), and 
that support for the LSC reforms and alignment of district policies increased as projects matured. 
Table 21 shows the predicted probabilities of a project receiving a particular rating at various 
time points. A project had a 22 percent chance of receiving a rating of Level 4 or Level 5 in its 
Baseline Year, compared to a 77 percent chance in its Final Year. 



Table 21 

Predicted Probabilities for the Continuum 
Rating of Extent of Support of the LSC Reforms 





Predicted probabilities* 


Baseline Year 


Year Two 


Final Year 


Level 1 : Predominance of Non-Supportive Context 


0.00 


0.00 


0.00 


Level 2: Exploring Supportive Context 


0.10 


0.04 


0.01 


Level 3: Transitioning Toward a Supportive Context 


0.68 


0.54 


0.23 


Level 4: Emerging Supportive Context 


0.21 


0.41 


0.70 


Level 5: Exemplary Supportive Context in Place 


0.01 


0.02 


0.07 



* Evaluator ratings significantly different over time, p < 0.05. 



Impact on Stakeholder Support 

Despite an array of barriers — administrative turnover, new initiatives, competing priorities — 
LSCs appear to have made progress in building stakeholder support for reform. Pis and 
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evaluators provided evidence in support of composite data on increasing levels of stakeholder 
support over time, in particular citing the adoption of new materials and frameworks aligned 
with the LSC, and the allocation of resources to sustain materials management centers as 
indicators of high levels of commitment to interventions initiated by the LSCs. 

• Principals 

LSC Pis typically cited principal support as the most important factor in determining teacher 
participation in professional development, and in developing a supportive context for reform at 
the building level. Even in those districts with a long history of supporting mathematics or 
science education prior to the LSC, Pis noted a deeper commitment among principals as a result 
of LSC efforts. Evaluators and Pis cited examples of principals who were active participants in 
professional development, and who looked for ways to support teacher leaders, budget new 
resources, create opportunities for teacher collaboration, and educate parents about new 
mathematics and science programs. 

A longitudinal analysis of principal questionnaire data (Simpson & Banilower, 2004) examined 
changes in three composite scores: (1) principal attitudes toward reform-oriented teaching (e.g., 
opinions on the importance of having students work in cooperative groups and participate in 
appropriate hands-on activities); (2) principal support for reform-oriented teaching (e.g., 
preparedness to support teachers in implementing current national standards, willingness to 
accept the noise that comes with an active classroom); and (3) the effect of resource availability 
on instruction (e.g., quality of available instructional materials, funds for purchasing equipment 
and supplies, time available for teachers to work with other teachers). The analysis found that in 
both mathematics and science projects, scores on these composites were quite high, even at the 
beginning of the LSC. Although statistically significant changes were found for some of the 
composite scores, most of the changes were not substantial. (See Table 22.) 



Table 22 



Effect of Project Year on Principa 


Questionnaire Composites 




Standardized Regression Coefficient 




Mathematics 


Science 


Attitudes Toward Teaching 


-0.09** 


-0.09* 


Principal Support 


-0.09 


0.03* 


Effects of Resource Availability 


0.08 


0.10* i 



* p < 0.05; ** p < 0.01 



Over time, LSCs came to better understand the roles that principals play in supporting reform. 
Initially, however, many LSCs were unclear about the level of attention needed in this area, and 
this sometimes affected the quality and impact of outreach efforts. Limited resources also 
influenced the level of attention LSCs devoted to principals, and as a result, projects at times 
neglected these stakeholders. In hindsight, some Pis noted their “flawed” approaches and early 
“oversights” in this area, and the consequences of their neglect. Lor example, one PI of a K-8 
mathematics/science LSC described early efforts to engage principals by linking them with lead 
teacher teams; project leaders then “took their eyes off’ principals, diverted by their efforts to 
provide professional development to teachers. In the last two years of the project, the LSC 
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refocused their attention on principals, convened a small group, “admitted” their error of 
omission, and gave principals recognition as “critical players.” The group then collaborated in 
developing a long-term plan for a professional development program for principals, based on 
their needs and priorities. 

Schools with stable leadership — those with the same principal since the beginning of the 
LSC — were significantly more likely to have positive outcomes on several measures (Crawford 
& Banilower, 2004). For all but one outcome, the distribution of predicted probabilities (i.e., the 
probability that a school will fall into a particular category) shifts toward the higher categories 
when the principal has been in the school since the beginning of the LSC. (See Tables 23 and 
24.) 



Table 23 



Effect of Principal Retention: Mathematics 





Predicted Probability 




Not Same 
Principal 


Same 

Principal 


Progress in moving towards excellence* 






Not ideal 


0.04 


0.03 


Beginning to improve 


0.53 


0.43 


Well along in improving 


0.37 


0.45 


Approaching ideal 


0.06 


0.08 


Achievement of students compared to 5 years ago* 






Much to somewhat worse 


0.09 


0.06 


About the same 


0.34 


0.27 


Somewhat improved 


0.48 


0.55 


Much improved 


0.08 


0.12 


Percent of teachers implementing the district-designated instructional materials* 






0-20 Percent 


0.42 


0.34 


30-50 Percent 


0.21 


0.21 


60-80 Percent 


0.21 


0.24 


90 or more Percent 


0.16 


0.21 


Percent of teachers involved in LSC-professional development activities* 






0-20 Percent 


0.48 


0.39 


30-50 Percent 


0.19 


0.20 


60-80 Percent 


0.17 


0.21 


90 or more Percent 


0.15 


0.21 



* Principal retention is a significant predictor of ratings on this indicator, p < 0.05. 
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Table 24 

Effect of Principal Retention; Science 





Predicted Probability 


Not Same 
Principal 


Same 

Principal 


Progress in moving towards excellence* 






Not ideal 


0.04 


0.03 


Beginning to improve 


0.44 


0.35 


Well along in improving 


0.43 


0.49 


Approaching ideal 


0.09 


0.13 


Achievement of students compared to five years ago* 






Much to somewhat worse 


0.03 


0.02 


About the same 


0.23 


0.17 


Somewhat improved 


0.57 


0.58 


Much improved 


0.17 


0.23 


Percent of teachers implementing the district-designated instructional materials* 






0-20 Percent 


0.14 


0.12 


30-50 Percent 


0.16 


0.15 


60-80 Percent 


0.30 


0.29 


90 or more Percent 


0.40 


0.44 



* Principal retention is a significant predictor of ratings on this indicator, p < 0.05. 



Several factors may explain the diminished impact of LSCs in schools with principal turnover. 
According to Pis, once principals came to understand the LSC’s message, they typically became 
more supportive; but as with teachers, principals needed time and exposure to new instructional 
practices to fully understand and support them. Turnover among principals reduced the amount 
of time LSCs had to work with new principals to gain their support. In addition, new principals 
may have felt little of the same level of commitment to LSC reforms as those who supported the 
program initially. 

Thus, while principal retention seems to have a positive effect on reform, principal turnover may 
have the opposite effect and reduce the measurable impact of the LSC program on schools and 
principals. As shown in Table 25, principal turnover was fairly high in LSC schools. Data 
collected from projects in Year Two show that 79 percent of the principals had been in the 
school since the beginning of the project; 86 percent had been in the district since the beginning 
of the project (Crawford & Banilower, 2004). Data collected from Final Year projects show that 
only 45 percent of principals had been at the school since the beginning of the project; 61 percent 
had been in the district since the beginning of the project. 
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Table 25 



Principal Retention Rates, by Data Collection Year 





Percent of Schools 




Year Two 


Final Year 


Overall 






School 


79 


45 


District 


86 


61 


Science 






School 


79 


42 


District 


87 


60 


Mathematics 






School 


78 


46 


District 


85 


61 



Within districts, a number of other barriers — instructional material adoptions that countered the 
LSC vision, high stakes tests that were not well-aligned with the LSC vision, and changes in 
district administrators also influenced the extent to which LSCs were able to fully engage 
principals. Securing the active support of principals, was a challenge for LSCs. According to 
some Pis, principals often provided “lip service” to reform, but no clear expectations for teachers 
to use the materials or participate in professional development, thus diminishing the potential 
impact of the LSC in the school. 

• District Administrators 

Across LSCs, efforts to gain the support of district administrators alternately met with great 
success and reoccurring setbacks. Where LSCs established strong working relationships with 
district administrators — including superintendents, school boards, curriculum directors, Title II 
staff, and others — Pis and evaluators noted that the potential for sustained support increased 
significantly. In one LSC, project leaders engaged superintendents from the eight participating 
districts as co-PIs, meeting with them three times a year, and involving them in key policy 
decisions under the LSC grant — for example, around the adoption of instructional materials and 
district allocation of staff development days. In turn, superintendents were integral to removing 
roadblocks and demonstrated their commitment by using general funds to adopt new materials 
after they were taken off the state adoption list. Other superintendents who attended national 
leadership institutes with LSC leaders often demonstrated high levels of commitment by 
promoting the adoption of designated instructional materials or mandating participation by 
teachers or principals in LSC professional development. 

Superintendents have further demonstrated their support by allocating funds to support science 
materials replenishment centers and providing resources for training new teachers. In some 
cases, superintendents have appointed other district personnel as the key contact persons, and 
these individuals have also been well positioned to influence resource allocation, professional 
development, and curriculum issues in support of the LSC. In fact, mid-level administrators 
often proved to be the strongest and most stable allies, and LSCs capitalized on the position and 
leadership of these individuals. Relatively low turnover among these administrators resulted in a 
loyal group that the LSC engaged in efforts to promote the LSC vision and to align LSC reforms 
with district priorities. 
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Challenges in developing support among district administrators included high turnover rates 
among superintendents, and the need for recurring efforts to make the case for reform with new 
administrators. Some Pis and evaluators cited repeated changes in superintendents, particularly 
in urban districts. The changes often resulted in new priorities, variable support, and mixed 
messages at the building level. In some cases, LSC leaders were unable to communicate their 
message well enough to build sustained support among district administrators for the adoption of 
designated instructional materials and professional development for teachers in mathematics/ 
science. For example, during the third year of one LSC, district commitment to standards-based 
instruction diminished considerably, as the district pushed other curriculum materials that were 
more aligned with the state assessment. The decision ultimately undermined the LSC vision, and 
LSC players were unable to deal effectively with the opposition. 

• The Broader Community 

Pis and evaluators suggest that, overall, LSCs have had mixed results in efforts to build 
community support. When parents participated in LSC activities, projects reported high levels of 
enthusiasm. Typically, however, LSCs reached only “small pockets” of parents, resulting in low 
levels of awareness. Pis sometimes noted that while project-wide parent outreach efforts worked 
well initially, participation diminished in subsequent years, and LSC staff shifted the 
responsibility for these activities to individual schools. Other Pis cited examples where teachers 
expended a high level of effort to conduct school-based parent activities, but few parents 
attended. Finally, some Pis noted that they had initiated parent activities, or planned to, but the 
work of professional development soon outweighed these efforts. 

LSCs’ experiences with developing support among teacher unions reflected that of parent 
outreach activities — a somewhat hands-off approach that resulted in neither much support nor 
much opposition to reform. There were examples of success, in which union leaders participated 
in the LSC planning process, promoted the program in union newsletters, supported the adoption 
of new materials, and shifted their positions to support more professional development in 
teachers’ contracts. Evaluators and Pis also described pockets of resistance; for example, in 
some cases, unions raised issues about time and compensation for teachers participating in LSC 
professional development outside the school day. 

LSCs have built “diverse” partnerships and “significant collaborations” with informal science 
partners, research institutions, and professional development centers. Some LSCs successfully 
formalized partnerships through the creation of university-based centers, with clear functions for 
sustaining reform — for example, in disseminating materials and professional development to 
both LSC and non-LSC districts. Said one evaluator about the role the project played in 
establishing partnerships that remained powerful forces in supporting reform: 

An important outcome of the project has been the development of the “partnership ” 
among the five districts and the university... Each partner has brought considerable local 
strengths to the table to share with the others; each has been willing to explore new ideas 
presented and to re-examine loccd practices that stood in the way of achieving their 
goals. As the grant ends, there is every indication that this partnership will endure. 
(Evaluator, K-8 science LSC) 
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Another evaluator made a similar point: 

An important and far-reaching strength of the LSC professional development is the 
infrastructure of relationships and connections the project has deliberately established to 
support professional development for teachers in particular, and the science 
improvement effort in general. They include relationships with all 14 school districts; the 
strong working relationship with the local university... as well as numerous relationships 
with groups and professional development providers around the United States. The 
achievement of establishing the current network of collaborations is a special highlight of 
the project, given the previous history where working relationships among these groups 
were noticeably lacking. (Evaluator, K-8 science LSC) 

LSCs have had considerable success in working with universities, resulting in the co-sponsorship 
of Inquiry Institutes; in-kind support for facilities and staffing of materials management centers; 
and the development of pre-service and in-service mathematics/science courses that hold 
potential for sustained professional growth experiences for teachers. Said these evaluators of 
cases in which LSCs successfully built university partnerships, and the implications for 
sustaining reform: 

Collaborations with the university have helped seal [an LSC] -infused philosophy as well 
as a strong working hand into the university’s science-related offerings. Together the 
university and [LSC] have designed courses and programs that reflect the LSC vision, 
and through their work with the teachers who seri’e the districts have helped steadily 
implement that vision in the classroom. (Evaluator, K-8 science LSC) 

The faculty and staff at the [university] have a long-standing relationship with the district 
and that relationship has continued to be a productive and cooperative one throughout 
the four years of the [LSC]. University faculty and mathematics educators are 
continuing to work with the district in the fifth year extension of the project, and plans 
are underway for continuing collaboration in state-funded professional development in 
mathematics education. (Evaluator, K-8 mathematics LSC) 

The college is moving to link its elementary science education program to the [LSC] 
project, using the LSC materials as a context for exploring standards and pedagogy in 
the science methods course. Elementary teachers graduating from [the college [ will 
have had experiences similar to teachers in the LSC summer institutes. Thus, as the 
districts encounter an expected wave of retirements in their aging teaching population, 
they will have access to new teachers ready to step into the program. (Evaluator, K-8 
science LSC) 

As with strategies to involve parents and teacher unions, however, LSC efforts to build 
partnerships were limited by time, resources, and the demands of the LSCs’ primary 
mission — teacher enhancement. 
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Impact on Policy Alignment 

The policy context in which LSCs operated was often a fickle one, influenced by administrative 
changes and conflicting priorities. Within this vacillating policy environment, LSCs were 
especially successful in aligning curriculum frameworks with the LSC vision, and in securing the 
widespread adoption of high-quality materials by schools and districts. These successes had 
major “systemic” implications, resulting in classroom changes by both LSC professional 
development participants and non-participants. 

Evaluators provided numerous examples in which LSCs played a crucial role in elevating the 
status of mathematics and science education, in aligning policies and procedures, and in pushing 
districts to take responsibility for sustaining these changes. Changes were noted by teachers and 
district administrators, as well as by LSC staff. For example: 

For me, the most important thing about what [the LSC] has been able to do over the 
years is to get science for elementary kids on the docket, the time devoted to science and 
then the quality of engagement. Ten years ago, I look back, and it wasn ’t there. . . 
[Science] is no longer a set-aside and it is not on the whim of curricular adoptions, kind 
of how the state used to be — oh, it is a science year. Every year is a science year for us 
and [now] it continues to be at the center.... It is embedded now in the culture. You have 
to pay attention to it. (Teacher, K-8 science LSC) 

[Everyone is] speaking the same language. Everyone understands the science agenda 
and is making sure schools are paying attention to it.... Now, that took a long time to 
create, and we are constantly working on keeping it going, but it is a powerful piece, or 
place where we are as a district. We think of ourselves as a coordinated system, serving 
the schools... and that is a whole different way that this system has aligned itself. 

(District administrator, K-8 science LSC) 

When [the LSC ] came along, it was almost like an abdication [for the district], like okay, 
[the LSC ] will take care of our science for us, and over the years, it has evolved to where 
now you can see the district starting to take over again the responsibility of where they 
are going in science... to having formal procedures and policies around science in the 
districts and they continue to support those sites that aren 7 participating in [the LSC] 
right now. They continue to support lead teachers, they continue to have district-wide 
meetings and talk about the direction they want to go as a district in science. . . That 
change has occurred in probably the last year and a half, two years. (LSC staff member, 
K-8 science LSC) 

Evaluators reported that LSCs played critical roles in promoting high-quality materials and 
standards-based instruction, and in clarifying ways in which the designated instructional 
materials could “deliver” in terms of addressing policies. One LSC — relatively unconcerned that 
state assessments were not aligned with the designated materials — focused activities on materials 
adoption, aligned with new district standards and LSC professional development. As the 
adoption got underway early in the LSC, teacher committees (with good representation from 
LSC-trained teachers) reviewed materials and made suggestions to the district. The district 
adopted the materials, established a materials resource center at that time, and developed a 
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strategic plan for science education that included district K-8 standards (aligned with the LSC 
vision), and professional development for new teachers. 

Other indicators of districts’ policy shifts included modifications in teacher hiring and evaluation 
procedures to reflect the LSC vision for teaching and learning. For example, one district revised 
its criteria for evaluating teachers to align expectations with LSC teaching practices and the 
designated materials, and based hiring decisions on applicants’ knowledge and experience with 
“reform math.” In another project, LSC efforts to work with the district Professional 
Development and Career Ladder Departments resulted in changes to vision and policy for 
teacher preparation and professional growth, and the adoption of new incentives for teacher 
participation in professional development. 

Other LSCs deliberately linked their efforts to college/university pre-service education programs 
to help ensure that the LSC vision for teaching gained a foothold in this setting. For example, in 
one project the LSC modules are now used in the university’s science methods course to 
examine both content and pedagogy. Given that most of the new teachers hired by the local 
districts receive their preparation through this pre-service program, the LSC’s efforts have 
helped ensure that teachers enter the field with some familiarity with the LSC vision and the 
designated materials, and that prospective teachers will continue to be exposed to aligned 
practices through the university’s elementary education program. 

Other Pis noted that, while the LSCs had had some influence on policy alignment, policies well 
beyond their control were more likely to influence district priorities and direction. Among these 
were state professional development funds linked to the adoption of instructional materials 
misaligned with the LSC vision, and certification requirements for professional development 
providers which conflicted with LSC strategies. As a result of these external factors, LSCs’ 
success in promoting aligned policies in support of the LSC vision varied widely. 



Development of Systems to Support Reform 

Evaluators and Pis collaborated on ratings regarding the extent to which districts had the 
capacity, infrastructure, and resources devoted to mathematics/science reform at various stages 
of the project (Baseline Year, Year Two, and Final Year), and predicted the likelihood that these 
systems would exist after LSC funding. These ratings were combined into three composites: 

• Systems for planning and implementing mathematics/science professional 
development (e.g., having structures in place for assessing teachers’ needs, having the 
capacity to plan and deliver high quality professional development, having incentives 
for teachers to participate in ongoing professional development); 

• Systems for aligning policies with the LSC reform vision (e.g., aligning curriculum 
frameworks, assessments, teacher evaluations); and 

• Systems for garnering and maintaining stakeholder support (e.g., from targeted 
teachers, principals, parents). 
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As shown in Figure 15, LSC districts typically did not have strong systems in place prior to the 
LSC (year zero) for any of the three purposes (Shimkus & Banilower, 2004a). However, 
districts were much more likely to have these systems in place by Year Two and to maintain 
them until their Final Year (typically, project year five). Between these two time-points, there 
was an average yearly gain of 0.31, 0.25, and 0.23 standard deviations respectively on the three 
composites. Although evaluators and Pis predicted a slight drop-off in these systems after LSC 
funding (coded as project year seven), many of the gains made during the LSC were expected to 
be sustained. 



District Systems Composites 




— -♦ — Systems for Professional Development* 

Systems for Aligning District Policies* 

— -A— - Systems for Garnering and Maintaining Stakeholder Support* 

' Project Year Seven data are predictions about future systems made by the project during its Final Year. 
* Composite score changes significantly over time, p < 0.05. 

Figure 15 



Infrastructure 

Evaluators and Pis cited some significant ways in which LSCs had influenced districts to support 
professional development in mathematics/science. In one urban district, for example, more 
strategic planning and coordination across central office departments resulted in a “shift” that 
provides stronger support for sustained professional development for both teachers and 
principals. Some LSCs successfully pushed for mandatory professional development for 
teachers. For example, one PI noted that districts now require three hours of professional 
development a month as part of the teacher’s contract. In another project involving multiple 
districts, an LSC-trained Teacher-in-Residence continues to conduct post-grant mandatory 6-10 
hour in-service sessions in science each year, resulting in a “steady state program” to address the 
needs of teachers. 

Among the most important structures initiated by LSCs were those to ensure professional 
development for teachers new to the instructional materials. In one LSC, for example, five 
districts jointly established a beginning teacher program to ensure training for newly hired 
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teachers. Other evaluators noted cases in which urban districts with high rates of turnover had 
institutionalized beginning teacher institutes with district support. In some cases, LSCs 
successfully sought external funding to support new teacher orientation sessions and mentoring 
programs; other projects mandated school-year sessions for beginning teachers. Some LSCs 
were able to influence the way districts viewed and supported ongoing professional development 
opportunities. These evaluators provided examples where this occurred: 

This [elementary] division is responsible for supervising and supporting the elementary 
school principals and in the past has seen themselves as being involved in only 
operations [and] not curriculum improvement. District administrators believe that the 
new direction of the elementary division is an indicator that the system is aligning itself 
to impact students and that this institutional structure is a critical piece in sustaining and 
coordinating the professional development for principals and teachers. (Evaluator, K-8 
science LSC) 

The concept of regular, high quality professional development in mathematics provided 
system-wide by the districts was new to [the district] when it was initiated in the first year 
of the LSC. Before that, professional development was a hit and miss situation, with 
teachers selecting their own activities for their own purposes. The consequence was a 
lack of consistency in curriculum and delivery of instruction from teacher to teacher, and 
school to school that has been eliminated with the introduction and institutionalization of 
a policy of district-supported, system-wide professional development.... The district’s 
commitment to continue grade -specific professional development when the NS F funding 
ends ensures the institutionalization of regular, quality professional development for 
mathematics teachers. (Evaluator, K-12 mathematics LSC) 

This closeness [of the LSC] with the district fosters and almost guarantees 
institutionalization of the project’s accomplishments. For example, the project is offering 
some of its professional development at the district’s required in-service training. This 
somewhat limits what can be offered, but at the same time moves science up in 
importance in the minds of the teachers and the district administrators. (Evaluator, K-8 
science LSC) 

In science LSCs that were ending their funded period, evaluators and Pis reported cases in which 
districts had allocated resources for maintaining materials resource centers. In these LSCs, 
contracts with the school district for operating the science materials center and for the 
renewal/refurbishment of the consumable items signaled continuing and tangible support for 
programs put in place by the LSC. Evaluators sometimes attributed the support among teachers 
for LSC reforms and the likelihood of sustainability to efficient and well-managed materials 
management systems initiated by LSCs. 

Evaluators noted other successes as projects took steps to transfer the organizational components 
of LSC professional development to a district or coalition of districts. Numerous LSCs 
successfully moved from NSL-funded professional development to a fee-for- service structure, 
with professional development provided by universities or non-profit organizations. Other 
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examples of LSC-initiated structures that remained in place after the grant included the 
following: 



• One K-8 science LSC tapped into the districts’ summer school structure as a strategy 
for providing sustained, ongoing, regional professional development for lead and 
typical teachers in a classroom-like setting, with built-in opportunities for coaching, 
discussion, and reflection. The five- week, summer school academy initiated by the 
LSC and now fully supported by the districts ensures a “continual professional 
development model that allows teachers to grow as professionals and develop skills 
and knowledge based on their needs.” 

• An LSC successfully developed a college-based Materials Resource Center that has 
provided a hub for professional development and materials dissemination. The five 
participating counties provide resources to cover costs for kit refurbishment, staffing, 
and transportation. As part of the adoption process, the districts committed to 
maintaining fiscal support for the center. The materials center has been critical to 
implementation, and appears to be well positioned to maintain a high level of support 
for teachers in sustaining use of the materials and some professional development for 
new teachers. 

• In a K-12 mathematics LSC, district-supported “Learning Walks” by principals 
initiated under the LSC provide an ongoing mechanism for monitoring classroom 
instruction and gauging teacher needs. Project staff developed checklists and ratings 
to help with this process, so that principals can make informed recommendations on 
teacher professional development needs. 

• In a K-8 mathematics/science LSC, the district adopted the LSC Special Presenter 
model that enables grade-level mathematics and science teachers to attend 
professional development during the school day. Said the PI: “It’s excellent 
professional development, a big chunk of time in the school setting, designed for that 
school, in a professional manner.” 

• A transition to district support for professional development resulted in a fee-for- 
service model for summer professional development, and LSC districts have already 
instituted this model with new teacher training. According to the PI, it is high quality, 
reasonably priced, and “highly leveraged” professional development that districts are 
more than willing to “ante-up” for, even in the worst economic downturn. 

• LSC leaders created a non-profit organization through which to operate professional 
development programs. The organization has proven to be a strong piece of 
infrastructure for sustaining LSC work. 

Evaluators also reported some concerns about the likelihood of institutionalization of high 
quality professional development. Not surprisingly, where they existed, professional 
development opportunities after the LSC grant had ended were far reduced from what had been 
offered during the life of the LSC. Some districts mandated that teachers participate in training 
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in order to receive the adopted materials, but these policies did not ensure ongoing participation 
over time to enhance content and pedagogical skills. Said these evaluators about some of the 
deterrents to the institutionalization of professional development: 

District-wide structures that support implementation of standards-based instruction on 
campuses and meaningful teacher collaboration have not been developed. (Evaluator, 
K-8 mathematics LSC) 

With fewer professional development sessions planned for 1999-2000, the lack of time 
and opportunities provided at the school-level for teachers to meet and plan during the 
school day becomes a more serious obstacle to reform than it has been in the past. 
(Evaluator, K-12 mathematics LSC) 

Both the lack of after-school and out-of-school time for professional development is an 
issue that is highlighted in many evaluation initiatives in [the state], not limited to the 
LSC. If the LSC hopes for institutionalization beyond the end of the project, teachers and 
school administrators must be aided in finding ways to balance the demands upon 
teacher time. Those initiatives that surx’ive will be those that become a part of a coherent 
model of school and classroom instructional improvement, (Evaluator, K-8 science 
LSC) 

Districts and schools need to reconsider school schedules and find or “make” time ( that 
is compensated) for teacher collaboration outside of class time. The lack of commitment 
to a continuing program for professional learning for teachers is of concern. (Evaluator, 
K-8 science LSC) 

LSCs sometimes underestimated what was needed to develop and sustain school-based 
professional development structures, and overestimated the capacity of project staff and/or lead 
teachers to do this work. Other evaluators noted the lack of formal district plans or teacher 
incentives to support participation in ongoing professional development beyond the grant. For 
example, the lack of stipends to support LSC-initiated action research groups in one district 
would likely mean the demise of these activities. As the LSCs ended, some states were facing 
enormous budget crises, leaving plans for continuing professional development “at the mercy of 
those kinds of processes in the system.” In addition, many LSC districts were financially 
“strapped” urban and rural school systems with “multiple areas of weakness,” thus resulting in 
fierce competition for funds. In more than one LSC, for example, professional development 
money had “dried up” and budget cuts had reduced key staff positions at the district level to 
support staff development and follow-up activities related to the instructional materials. 

Developing Capacity 

According to Pis and evaluators, LSCs leave behind an essential component for sustaining their 
interventions: teacher leaders. Teachers on special assignment and school-based teacher leaders 
often assumed active roles on school and district committees during the LSC, and according to 
Pis and evaluators, many have continued in these roles after the grant. Pis frequently attributed 
major project successes to teacher leaders, including efforts to align district curriculum with state 
and national standards, adopt high-quality instructional materials, and develop aligned 
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assessments. Evaluators reported that teacher leaders’ participation on reform-oriented 
committees helped broaden their understanding of district policies and practices, and provided 
them with a new perspective on how change happens. Said one evaluator: “[The teacher 
leaders] are a dynamic group that is likely to influence policies and practices for years to come.” 

In a number of LSC projects, support for teacher leaders was ongoing and of high quality, with 
planning, capacity building and professional development delivery intertwined. In some cases, 
these projects produced a savvy group of teacher leaders, who were closely involved in planning 
and delivering interventions, and who had accumulated many hours of professional development 
to prepare them for leadership roles — both within the LSC and outside of it. Some of these 
leaders moved into leadership positions in schools and districts, as well as regional and state- 
level positions, working in the areas of curriculum, professional development, and assessment. 

There was also evidence to suggest that districts, to varying levels, had bought in to the need to 
sustain these leaders beyond the LSC grant. Lor example, one district had contracted with LSC- 
trained teacher leaders to provide professional development to every K-12 teacher after the LSC 
grant ended. Another urban district had provided permanent funding for LSC teacher leader 
positions, resulting in “math coaches” released full-time in every school. Said the PI: “That is a 
structure the LSC put into place, and the district has copied it in all 97 buildings.” Pis and 
evaluators provided numerous other examples of the teacher leadership capacity that remains in 
the districts, and the implications for sustaining professional development and curriculum 
reform: 



The [Teacher] leaders in math education hold a unique position within the [district]. 

They stand poised to move the district’s math achievement beyond SAT 9-focused 
expectations. The knowledge and skills gained through their work with [ the LSC] have 
given them a level of expertise not known within the district before.... Their continuing 
presence is a major force in the long-term sustainability of the LSC reforms. (Evaluator, 
K-8 mathematics LSC) 

[The LSC helped] build district capacity for reform by developing large staffs of well- 
trained teachers in each target district. These cadres include teachers who are capable 
of providing professional development, revising curriculum effectively, mentoring new 
teachers, establishing science study teams, implementing extended school-day programs 
in science, and coaching experienced teachers. (Evaluator, K-8 science LSC) 

The districts are beginning to realize that they have a treasure trove of well-trained 
teachers who can help them sustain the growth of the project... I think this is where we’re 
going to have some lasting effects, is in these people who have recdly grabbed on to it and 
become, in the true sense, leaders. (PI, 6-12 mathematics LSC) 

Through [the LSC], as well as the earlier [project], the districts have developed a cadre 
of teachers who embrace the project ’s vision, can operationalize it in the classroom, and 
are interested in taking more of a leadership role for science program improvement. 
(Evaluator, K-8 science LSC) 
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Among the other examples provided by evaluators: 

• In a K-8 mathematics/science LSC, the district will continue to support four positions 
formerly funded by the LSC around diversity, technology, and mathematics 
professional development. 

• In a K-8 science LSC, 13 of 16 LSC districts picked up the cost for maintaining 
teacher leader positions formerly funded by the LSC; teacher leaders will assess needs, 
facilitate professional development, and assist with materials management. 

• In a secondary mathematics LSC, district funds support teams of teacher leaders to 
assess needs and plan appropriate professional development. The “math team” 
provides input to the superintendent on curriculum, assessment, and professional 
development. It is the only subject-specific team in the district. 

• In a K-8 science LSC, “Alumni Institutes” and mentoring programs funded beyond 
the LSC through district, university, and external funds will continue to increase 
capacity among teachers and teacher leaders. 

• Instructional Resource Teachers developed under the LSC have continued to conduct 
mathematics sessions of district-required summer professional development for new 
teachers. For two years after the LSC ended, the district supported about 60 of these 
leaders as Mathematics Coaches — about one for every two schools. 

All of these examples suggest that some LSCs were able to build a sense of appreciation for 
teacher leaders, and secure district support for their continuing roles as professional development 
providers. On the other hand, teacher leaders progressed at a rate slower than LSC staff had 
anticipated — evidence of a “time lag” between capacity building efforts and teachers’ ability to 
assimilate and become competent enough to put into practice what they were learning. Thus, 
while the LSCs moved teachers in the direction of stronger leadership and capacity to support 
reform, they also fell short in their expectations for having consistent leadership in place in LSC 
schools. In addition, according to one PI, “It’s hard for urban districts to understand that part of 
sustaining reform is not just professional development for new teachers, but professional 
development for your lead teachers.” Without a mechanism to sustain support for leadership 
development, there was some question as to how lead teachers trained under the LSC might 
continue to function effectively as leaders. 



Likelihood of Institutionalization of LSC Reforms 

At the end of each project’s Year Two, and again in the Final Year, the evaluator and PI were 
asked to provide an overall rating of the likelihood of institutionalization of the LSC reforms 
using a scale ranging from “Level 1: Likely to snap back at the end of the LSC” to “Level 5: 
Long term institutionalization is likely.” Shimkus and Banilower (2004a) found that by Year 
Two, districts in over 70 percent of projects submitting ratings were expected to institutionalize 
many, if not all, of the LSC reforms. (See Table 26.) The fact that the ratings at the two time 
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points are not statistically different is an indicator that projects continued to have those 
expectations as NSF funding was drawing to an end. 15 



Table 26 



Continuum Ratings of Likelihood of Institutionalization of LSC Reforms 





Percent of Projects 


Year Two 
(N = 58) 


Final Year 
(N = 61) 


Level 1 : Rubber Band Likely to Snap Back 


0 


0 


Level 2: LSC Reforms Likely to Gradually Fade Away 


0 


2 


Level 3: Minor Components Likely to Become Institutionalized 


30 


19 


Level 4: Components Likely to Become Institutionalized 


63 


62 


Level 5: Institutionalization of LSC Reforms Likely 


8 


17 



As noted earlier, however, even Pis who had been able to secure sustained support for 
mathematics and science education spoke of the challenges of institutionalizing reforms in an 
erratic policy environment. For example, said one PI about the fragile state of the district’s 
materials management system, due to a proposed tax cap which would have eliminated the 30- 
year-old center: “We could have lost everything. So yes, it’s institutionalized. . .but it’s always 
at the whim of state or municipal funding.” In another LSC, changes in the state assessment 
away from inquiry-based learning and toward basic skills had a considerable impact on the 
degree to which schools and districts were inclined to sustain their use of the designated 
instructional materials: administrators and parents saw the program as a riskier venture, given 
the new policy environment, and LSC staff were unable to make a strong enough case to 
overcome this resistance. Finally, some Pis (particularly those focused on science) noted that, by 
the end of the LSC, districts were ready to move their attention to other subjects, or initiatives. 
These Pis described the challenges this way: 

The science program will have to compete with other instructional needs for both time 
and money. At the moment, the community and the central administration are 
comfortable with science. They now see elementary school science as a core subject area 
without any major problems. Unfortunately, this does not give science a competitive 
advantage in gaining resources. Presently, reading and mathematics are the “hot 
button ” areas of the curriculum, as they are nation-wide. The district is currently 
funding efforts to remedy problems in reading and math. (PI, K-8 science LSC) 

The biggest challenge is the landscape of literacy and mathematics which have the 
highest priority of anything anywhere, and the idea that science takes a low priority to 
those, especicdly over the last two to three years. When we started, districts funded 
school-based facilitators [for science] in every school. When the grant ended, those 
positions were removed and replaced by facilitators for literacy and math. (PI, K-8 
science LSC) 



15 In longitudinal analyses, multilevel modeling can incorporate cases with incomplete data to improve estimation of 
values at each time point. 
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One barrier to institutionalization is the concern of district administrators regarding the 
number of competing initiatives. There is pressure for change coming form multiple 
sources and the teachers have a limited amount of time to work together. There are state 
initiatives and district initiatives. . .Science education is only one of many items that 
demand nonexistent funding. (PI, K-8 science LSC) 

Another barrier to institutionalizing LSC reforms related to the LSCs themselves. Pis often 
lacked the time, experience or vision for building “institutionalization” activities into the project 
design. The background of LSC leaders sometimes left them better prepared to design, manage, 
and implement teacher enhancement activities than to deal with issues related to policy and 
systems. Thus, while some Pis advised knowing what the system will bear, leveraging support, 
and building in “transitioning” mechanisms to ensure institutionalization, others admitted to not 
having a real sense of the level of work required. In some cases, LSCs’ limited understanding of 
the scope of work and level of commitment needed had huge implications for moving beyond 
teacher enhancement to systemic reform and the institutionalization of policies and practices 
needed to sustain LSC activities. 
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Part Six 



Summary and Implications 



Since 1995, NSF’s Local Systemic Change program has been supporting schools and districts in 
their efforts to improve mathematics, science, and technology education. Data from the LSC 
core evaluation suggest that the initiative has had significant success in a number of areas. The 
data also provide some insights into the difficulties that large-scale reform projects encounter in 
their efforts to provide professional development for substantial numbers of teachers, and to 
develop a supportive context for scaling up and sustaining interventions initiated with external 
funds. This section looks at some of the LSC’s key successes and challenges, and the 
implications of the LSC experience for others engaged in the reform of mathematics and science 
education. 



Major Successes 

High Quality Professional Development 

The LSC’s primary goal was to provide teachers with high quality professional development 
around mathematics, science, and technology education. Core evaluation data suggest that 
projects have done well in this area. Evaluators’ ratings for the overall quality of professional 
development programs were high along a number of dimensions. In particular, LSCs were 
strong in creating a culture conducive to teacher learning, in the quality of preparation of 
professional development providers, and in preparing teachers to use high-quality materials and 
appropriate pedagogy in their classrooms. Further, ratings for overall program quality and the 
quality of individual sessions improved significantly over time, suggesting that LSCs became 
more adept at implementing high quality professional development as they matured. 

Impact on Teachers and Teaching 

Data collected as part of the core evaluation suggest that LSC professional development had a 
positive impact on teaching and learning. Teachers’ attitudes toward reform-oriented teaching in 
mathematics and science, and their perceptions of their content and pedagogical preparedness to 
teach these subjects improved over time with increased participation in LSC professional 
development. Data from teacher questionnaires and evaluator ratings of classroom observations 
also suggest that teachers’ participation in LSC professional development is linked to a number 
of positive outcomes in their instruction, including: (1) overall improvement in the quality of 
mathematics/science lessons; (2) increased time spent on science instruction in the elementary 
grades; (3) enhanced quality of content presented to students; (4) more frequent use of 
investigative practices, questioning, and sense-making practices; and (5) a greater likelihood that 
the classroom culture promotes intellectual rigor and student engagement. Core evaluation data 
also suggest that the quality of classroom instruction improves over time as teachers’ 
professional development hours accrue, although there seems to be a limited impact beyond 80 
hours of professional development. These impacts were found regardless of teachers’ collegiate 
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mathematics/science preparation, and helped narrow the initial differences between teachers with 
strong and weak content preparation on teacher perceptions of their content and pedagogical 
preparedness composites. 

Widespread Use of High-Quality Materials 

Efforts to increase teachers’ use of high-quality instructional materials in mathematics/science 
have been a cornerstone of the LSC program. The widespread use of these materials in the 
classroom by participating teachers points to success in achieving this objective. LSC 
professional development focused heavily on providing teachers with opportunities to explore 
the materials through practice, investigation, problem-solving, and discussion — strategies that 
likely contributed to teachers’ tendency to use the materials with their students. Core evaluation 
data suggest that the quality of instruction improved with teachers’ use of the designated 
materials, and that teachers were more likely to use these materials as they accumulated LSC 
professional development hours. The LSC also played a major role in ensuring that the 
designated instructional materials became a fixture in the educational landscape in participating 
districts by promoting the formal adoption of these materials. These efforts appear to have had a 
“systemic” effect on both participating and non-participating teachers in LSC districts, resulting 
in the increased use of these materials as LSC projects mature. 

Supportive Context for Reform 

LSCs report a steady increase in support for reform over time, suggesting that projects have had 
success in building stakeholder and policy support for the LSC vision and interventions. Ratings 
by evaluators and Pis suggest that while LSCs began their projects in an environment leaning 
toward reform, participating districts were less likely to have systems in place prior to the LSC to 
support reforms. In contrast, districts were much more likely to have these systems in place in 
the last year of LSC projects, suggesting a link between project efforts to build a supportive 
context for reform and the development and institutionalization of systems for maintaining 
professional development, as well as stakeholder and policy support. The adoption of the 
designated instructional materials and aligned frameworks, the creation of science materials 
management centers, districts’ allocation of funds to support teacher leaders in varying 
capacities, the development of partnerships with universities, non-profit organizations and others 
to support continuing professional development — all provide evidence of the success LSCs have 
had in building capacity and infrastructure to support and sustain reform. 



Major Challenges and Implications 

Preparing and Supporting Professional Development Providers 
While LSCs typically received high ratings in the quality of preparation of professional 
development providers, evaluators also pointed to some areas of weakness among providers 
which sometimes limited the impact of professional development. These weaknesses related to 
both content sessions (e.g., lack of rigor) and sessions on pedagogy (e.g., lack of explicit 
discussion of strategies). Even in projects that were highly attentive to the preparation and 
support of professional development providers, session quality sometimes suffered due to the 
limited skills of some providers. In part, the need for LSCs to press professional development 
providers into service before they were fully prepared for their roles helps explain some of these 
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limitations. In addition, finite resources often limited the time and effort LSCs devoted to 
preparing professional development providers, as opposed to training “typical” teachers. 

Evaluators and Pis suggest the need for a sustained level of effort around preparing and 
supporting professional development providers, especially in areas that pose particular 
difficulties for teachers (e.g., content, questioning, closure). Projects might provide more 
professional development to providers around the mathematics/science concepts underlying the 
student modules; provide more emphasis on the demonstration and explicit discussion of 
questioning strategies; provide more illustrations of student-centered instruction and ways to 
promote high quality student-to-student interactions; and present effective ways to wrap up 
activities or lessons that help students make conceptual connections. 

Addressing Teachers’ Content Needs 

Among the dimensions of LSC professional development, deepening teachers’ understanding of 
mathematics/science content has been the most challenging for LSCs. In part, the attention that 
projects place on the designated instructional materials and on the mechanics of the modules/kits 
has reduced the LSCs’ ability to address teachers’ broader content needs. In addition, as noted 
throughout this report, professional development providers, particularly teacher leaders, may be 
under-prepared in content and lack the skills needed for helping teachers make sense of 
mathematics/science content in the context of both the activities and the broader curriculum. 
Linally, professional development sessions designed to deepen content knowledge and support 
teachers’ content needs during implementation sometimes failed to delve into the very content 
they were designed to address, due to other more pressing teacher concerns such as materials 
management or pedagogy. 

Each of these difficulties might be addressed with better preparation and support for the 
professional development providers charged with deepening teachers’ content knowledge. In 
addition, core evaluation data suggest that LSC professional development dedicates somewhat 
less time to content than pedagogy and instructional materials. To better address teachers’ 
content needs, projects might consider devoting a more equitable share of professional 
development time to content, and/or creating more content- specific sessions to meet teachers’ 
wide-ranging needs in this area. 

Supporting Teachers 

As with content, supporting teachers during implantation was one of the dimensions of 
professional development that posed challenges for LSCs. While projects fairly well excelled at 
supporting teachers in terms of materials and supplies needed for the implementation of the 
modules/kits, LSCs devoted far less attention to providing ongoing individual and small group 
support, in part due to the lack of resources and staffing for these kinds of activities. Given 
teachers’ lack of time and scheduling constraints, LSCs also encountered difficulty in recruiting 
teachers for participation in follow-up and support activities — for example, communities of 
learners designed to foster reflection around content, pedagogy, and materials. Teacher leaders, 
who were typically charged with providing ongoing support, were often spread thin and/or 
lacked coaching and facilitation skills needed for engaging teachers effectively. Evaluators also 
suggest that the lack of school support and/or the lack of formal structures limited these kinds of 
opportunities during the LSCs, and pointed to difficulties in sustaining support for teachers 
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beyond the grant. These barriers suggest the need for cultivating greater awareness among 
district and building-level administrators who are in positions to encourage teacher participation 
in ongoing professional development activities, and provide opportunities for collegial 
interaction, discussion, and mutual support among teachers engaged in reform. 

Engaging Principals 

In many ways, principals play key roles in determining the outcomes of LSCs — from 
encouraging teachers to participate in professional development, to supporting teachers’ use of 
high-quality materials and inquiry-based practices, to enabling the work of teacher leaders, to 
making time for teachers’ to participate in site-based professional development. Core evaluation 
data point to the significance of stable administrative leadership in a school, particularly in 
predicting the likelihood of teachers using the designated instructional materials. LSCs have 
struggled with balancing resources and level of effort devoted to teacher professional 
development as opposed to reaching out to key stakeholders. Attention to principals was 
sometimes an afterthought or was neglected altogether. Projects clearly have little control over 
some barriers that affect the extent of principal support (such as turnover or changes in district 
policies). However, given the key roles principals play in supporting reform and the fairly high 
rate of turnover, projects might do well to devise deliberate strategies for communicating their 
message to administrators, and tapping into existing structures for this purpose. Engaging 
principals in ongoing and fairly intensive ways, and making the case for reform with new 
administrators, seem to be worthy activities with potentially large pay-offs — both during and 
beyond the life of a major initiative. 

Reaching Targeted Teachers 

Despite efforts to provide a range of opportunities for participation, few LSC projects achieved 
the goal of reaching all eligible teachers with 130 hours of professional development. Teacher 
turnover, changes in state/district policies that reduced the number of professional development 
days available or shifted the focus to other subject areas, teacher reluctance to participate — all 
contributed to the problems LSCs encountered in reaching their targeted audience. In some 
districts, the large influx of new teachers, as well as mobility across schools and subject areas, 
hindered LSC efforts to meet the 130-hour goal. LSCs that underestimated teacher turnover — 
and many did — were forced to devise new strategies “on the fly” to meet the needs of new 
teachers, as well as those they had initially planned for. The implications for others engaged in 
similar kinds of large-scale reform efforts seem obvious — expect teacher turnover and build in a 
plan that includes new teachers and incentives for participation. Beyond this, projects need to 
look for more effective strategies to make the case for reform with both teachers and 
administrators to increase the likelihood of participation, and continue to make professional 
development relevant, accessible, practical, and of high quality to attract even those resistant to 
the reform vision for mathematics/science instruction. 
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